{"id":"a143e282-968c-4645-a3ad-5edb0907d901","arxiv_id":"2507.12367","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 328-problem execution-based benchmark shows the best AI code generators pass version-specific hidden tests only about half the time.","lead":"GitChameleon 2.0 adds a benchmark of 328 Python coding tasks, each requiring code that works with one specific version of a library. Across many AI coding systems, including agents and assistants, hidden-test pass rates stay near 50 percent for the strongest enterprise models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 48-51% success rates rest entirely on hidden tests generated by an AI agent with no independent human audit; a biased test instrument would invalidate all comparisons.","rationale":"The central claim is that current LLM-based systems achieve only about half success on version-conditioned code generation (Table 1, abstract). This claim depends on the hidden test suites being correct discriminators of version compliance. The reader's weakest assumption correctly identifies this. I agree: the tests were generated by Zencoder (GPT-4.1-based), with execution feedback and manual patching, but no independent human audit is reported (Appendix A.4). Coverage of 96.5% and the fact that reference solutions pass are necessary conditions, not sufficient ones; they do not ensure that tests reject solutions that use a deprecated API or accept all valid alternatives. Since models cluster within a few points (48-51%), a small systematic test error could change the headline. I found no internal mathematical error; the execution harness, dual version control (prompt plus pinned environment), and confidence intervals are sound. The absence of a human baseline and the in-distribution assumption for closed models are real but secondary limitations; a test audit is the single check that would settle the main concern. If the audit passes, the CONDITIONAL verdict should move to ACCEPT. If it fails, the headline numbers would need revision, and the verdict would move to REJECT or UNVERDICTED. The Appendix H knowledge-retention claim rests on a crude proxy (logic-node counts), but even if that characterization is imperfect, the benchmark's empirical finding would still stand as a measure of functional correctness on these tasks. Therefore my read does not change the reader's verdict.","tokens_in":36733,"tokens_out":5945,"duration_ms":65613,"concrete_test":"Independently audit a random sample of 50 (15%) of the 328 hidden test suites by two human Python experts who are blind to the reference solutions. For each suite, verify (i) the reference solution passes in the pinned environment, (ii) each test asserts the version-specific behavior named in the problem statement (e.g., using the deprecated API must fail, the changed argument name must be accepted, dtype/return-type matches the target version), and (iii) at least one deliberately incorrect solution (using the pre-change or post-change API as appropriate) fails. Record the fraction of suites with at least one incorrect pass/fail label. If that fraction exceeds 3%, recompute the headline hidden-success rates with the corrected labels; a shift larger than the reported ±2.8 standard errors would invalidate the 48-51% claim and the model ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that enterprise models achieve only 48-51% hidden-test success (Table 1) and that self-debugging, RAG, and agents improve but do not solve the task—is measured exclusively against hidden test suites constructed by the Zencoder agent (Appendix A.4). The paper reports 96.5% coverage and that reference solutions pass, but neither property establishes that the tests are correct discriminators. Coverage measures how much of the reference solution's code is executed, not whether tests assert the version-specific behavior described in the problem statement; and a reference solution passing its own tests is necessary but not sufficient for test validity. Because Zencoder is itself an LLM (GPT-4.1-based) with internet search, the tests can encode the same API misconceptions that the benchmark aims to detect (e.g., using a deprecated argument in the reference implementation, or failing to check return types/dtypes that changed across versions). Manual patches were applied, but the appendix does not report an independent human audit of pass/fail labels, nor any adversarial validation against known-wrong solutions (e.g., solutions using the pre-change API). If even a small fraction of the 328 suites mislabel correct solutions as failing or incorrect solutions as passing, the headline percentages and all model comparisons (Tables 1-4, Figures 6-8) shift; because models cluster within a few points of one another, a 3-5% systematic labeling error could reorder the ranking and change the 'significant challenges' conclusion. This is the load-bearing measurement assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GitChameleon 2.0 introduces a dataset of 328 Python code-completion problems, each tied to a specific library version and a documented breaking change, with executable hidden tests and a visible test per problem. The paper evaluates a broad range of open-weight and enterprise LLMs, tool-calling agents, CLI/IDE coding assistants, and RAG pipelines under greedy decoding, chain-of-thought, self-debugging, multi-step agent, and retrieval-augmented settings. The central finding is that enterprise models achieve only 48-51% hidden-test success under greedy decoding, with self-debugging and RAG providing gains yet leaving over 40% of problems unsolved. The paper also reports analyses by API change category, error type, and version release year, and argues that the benchmark primarily measures version-specific knowledge retention rather than complex logic.","tokens_in":36948,"tokens_out":4467,"duration_ms":46689,"significance":"If the hidden tests are valid, this is a useful benchmark and empirical study. Strengths include execution-based evaluation in Docker containers with pinned library versions, standard errors on all headline numbers, a wide model/agent/assistant coverage, public dataset and code, and the visible/hidden test gap analysis that supports the self-debugging findings. The moderate correlations with SWE-bench and LiveCodeBench are thought-provoking. The main weakness is that the hidden tests were AI-generated without independent verification, and the knowledge-retention interpretation rests on a subjective AST classification; these need to be addressed before the headline claims can be fully trusted.","major_comments":[{"comment":"The benchmark's headline metric is hidden-test success, yet the hidden tests were generated by the Zencoder agent (built on GPT-4.1) with no independent human audit; the reported 96.5% coverage and the fact that reference solutions pass do not establish that the tests correctly discriminate version-correct from version-incorrect solutions. Coverage measures how much of the reference implementation is exercised, not whether the assertions encode the version-specific behavior described in each problem statement, and a reference solution passing its own tests is necessary but not sufficient for test validity. Because Zencoder is itself an LLM, the tests can encode the same API misconceptions the benchmark aims to detect (e.g., using a deprecated argument in the reference implementation or failing to check return types that changed across versions). I request an independent human audit of a sample of hidden tests, or at least an adversarial validation in which known version-incorrect solutions (e.g., using the pre-change API) are shown to fail, and a report of the resulting precision/recall of the test suite. Without this, the 48-51% enterprise rates in Table 1 and all downstream comparisons rest on an unvalidated measuring instrument.","section":"Appendix A.4 (Tables 1-4)"},{"comment":"The claim that GitChameleon 2.0 'is primarily designed to assess version-specific knowledge retention rather than complex logic-based code generation' rests on an ad-hoc AST node classification whose categories are not independently validated. Table 15 counts 'calling a math or utility function with non-obvious purpose' and 'composing multiple calls together' as logic-related, yet these categories require judgment; no inter-annotator agreement or comparison with a validated complexity measure (e.g., cyclomatic complexity or human ratings) is provided. Since this claim is used to interpret the benchmark's results and to position the contribution relative to logic-reasoning benchmarks, it should be supported by a more rigorous validation, or the wording should be softened to describe the observed distribution of logic nodes without asserting the benchmark's primary design intent.","section":"Appendix H (Table 15)"}],"minor_comments":[{"comment":"The function name 'custom_violinpolot' in the figure appears to be a typo; it should probably be 'custom_violinplot'.","section":"Figure 1"},{"comment":"The code listing shows 'from scipy import llinalg' with a double 'l'; this is likely a typo for 'linalg'.","section":"Appendix F.2"},{"comment":"Cline with GPT-4.1 and Cline with GPT-4.1-nano report identical hidden success rates (54.6%) and identical API hit rates (48.8%); please verify whether this is a copy-paste error or an actual duplicate result.","section":"Table 3"},{"comment":"The Kymatio entry lists a librosa documentation URL instead of the Kymatio project URL; please correct this citation.","section":"Appendix J.1"},{"comment":"Capitalization of 'Zencoder' is inconsistent; the text uses both 'Zencoder' and 'ZENCODER'.","section":"Appendix A.4"},{"comment":"The entry 'LLM-Deprecated-APl' contains a typo; 'APl' should be 'API'.","section":"Table 14"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an empirical software engineering or ML evaluation venue. The dataset and code release are commendable. The main risk is the unvalidated hidden-test instrument; I would request an audit or adversarial validation before acceptance. I do not see grounds for rejection if the test quality is confirmed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GitChameleon 2.0 is a solid, reusable benchmark: 328 version-conditioned Python problems with pinned environments, executable tests, and a broad evaluation across greedy decoding, CoT, self-debugging, RAG, agents, and IDE/CLI assistants. The headline result — top models at 48–51% hidden-test success — is grounded in real execution against pinned library versions, with standard errors reported. The paper also does the field a service by publishing prompts, case studies, and a breakdown of failures by error type and by change category. Much of the analysis (e.g., visible-hidden gap widening after self-debugging, RAG lift being modest) is informative and reproducible.\n\nWhat's genuinely new: the jump from GitChameleon's 116 problems to 328, the AI-generated hidden test suites, and the expansion from plain decoding to agents and assistants. The dataset fills a gap next to SWE-Bench and LiveCodeBench: it tests version-conditioned knowledge rather than open-ended bug fixing or algorithmic problem solving. The appendix's logic-node analysis supports their claim that most problems test API recall rather than complex logic.\n\nThe main soft spot is the measuring instrument. Hidden tests were generated by Zencoder (GPT-4.1-based), and while the authors report 96.5% coverage and say reference solutions pass, they do not report an independent human audit of pass/fail labels or adversarial validation against known-wrong solutions. That is a legitimate concern: if even a few test suites encode the generator's own API mistakes, the absolute rates and tight rankings among models could shift. Note, however, that the paper explicitly acknowledges the construction in Appendix A.4 and does not hide it. The overall \"models struggle\" conclusion would likely survive moderate test noise — 50% is far from ceiling — but the precise ordering of models within a few points should be read cautiously.\n\nAlso worth noting: there is no human baseline (stated in the Limitations), and the claim that all closed models have seen the target versions in training is unverifiable. Both are minor, and the second is reasonable for a benchmark designed to test ID version control.\n\nWho this is for: anyone building or evaluating code generation models, RAG pipelines, or coding assistants. It deserves a serious referee; I'd accept it with a request for a test audit (human spot-checks or known-wrong-solution validation) and, if feasible, a small human baseline. The benchmark itself is the contribution, and it's a useful one.","headline":"A genuine, reusable version-conditioned benchmark with execution-based evaluation; the AI-generated test audit is the main thing to fix, but the central result is solid.","tokens_in":37579,"tokens_out":2271,"would_cite":true,"duration_ms":25538,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Best AI coders pass only half of version-locked Python tasks.","keywords":["GitChameleon 2.0","version-conditioned code generation","library version incompatibility","breaking changes","Python code benchmark","LLM code evaluation","self-debugging","retrieval-augmented generation"],"falsifier":"Have human experts write independent oracle tests for a random sample of the 328 problems, rerun the same models against those tests, and compare pass rates with the reported hidden-test rates; a material divergence would show the benchmark's difficulty depends on the AI-generated test suite. Then classify each wrong solution as either a version-wrong API call or a logic error under the correct version; if most failures are logic errors, the paper's knowledge-retention claim would not hold.","tokens_in":36477,"feed_emoji":"🐍","tokens_out":9104,"duration_ms":98169,"temperature":0.7,"pith_summary":"GitChameleon 2.0 aims to establish that generating code for a specific, pinned library version is a distinct capability that today's best AI code generators do not yet have. It is a benchmark of 328 Python completion problems built from documented breaking changes in 26 popular libraries, where each prompt names a version and each solution is graded by hidden unit tests that the model never sees. Across enterprise models, agents, coding assistants, and retrieval-augmented pipelines, hidden-test success clusters near 48-51% for the strongest models, and open-weight and smaller models fall further behind. The failures are mostly attributed to recalling the wrong version's API rather than to logic, since most ground-truth solutions contain fewer than five logic-related AST nodes. If the benchmark is right, version-aware code generation is a real bottleneck for production use, and progress will need evaluation and training that reward knowing which API belongs to which version.","feed_headline":"Best AI coders pass only half of version-locked Python tasks","feed_subtitle":"A 328-problem benchmark pins library versions; top models score 48-51% even with docs and self-debugging.","key_machinery":"The central object is GitChameleon 2.0: 328 Python code-completion tasks, each tied to a documented breaking change in one of 26 libraries, with the library and exact version named in the prompt and installed in the validation environment. The key design move is version-conditioned generation (VCG), which forces models to write code for a fixed, in-distribution version rather than migrate code forward; the hidden-test suite is the scoring instrument, with visible tests reserved for self-debugging and reference documents reserved for RAG experiments. A supporting mechanism is the logic-vs-knowledge decomposition, which counts logic-related AST nodes in ground-truth solutions and uses the low counts to argue that the benchmark measures knowledge retention, not complex code logic.","core_discovery":"The paper's central claim is that state-of-the-art LLM-based coding systems are unreliable when asked to write code for a library version they have already seen. Even with the target version stated in the prompt and installed in the test environment, the best enterprise model reaches 51.2% hidden-test success and several large models cluster at 48-51%; the best retrieval-augmented result is about 59%, so more than 40% of problems remain unsolved even with documentation available. The benchmark deliberately uses versions inside training windows and mostly short solutions with few logic nodes, so it interprets the failures as a version-knowledge disambiguation problem rather than a code-reasoning problem. The paper's conclusion is that version-conditioned code generation is a measurable, currently unsolved subproblem of code generation, and that execution-based benchmarks are needed to see it.","pith_inferences":["Editorial extension: because the benchmark pins versions inside training windows, the 48-51% ceiling suggests pretraining stores versioned API knowledge in a poorly disentangled way; a targeted error analysis distinguishing stale-API calls from hallucinated newer APIs would make this precise.","Editorial extension: the hidden tests were generated by an AI agent and not independently human-audited, so auditing a random subset with human-written oracle tests would show how much of the reported difficulty is the test suite itself.","Editorial extension: since most ground-truth solutions have few logic nodes, an oracle-document RAG condition would isolate whether remaining failures come from retrieval inaccuracy or from models ignoring correctly retrieved version-specific documentation.","Editorial extension: the paper leaves out human baselines and version-to-version translation; adding a human-performer baseline would turn the absolute 48-51% numbers into a calibrated statement about how hard version-conditioned generation is for humans versus models."],"forward_implications":["Teams that ask an LLM to produce code for a pinned library version can expect roughly one in two completions to fail hidden functional tests, so version-aware guardrails and testing are necessary before such output can be shipped.","Providing relevant documentation through RAG lifts the best model to about 59% hidden-test success, yet leaves over 40% of problems unsolved, showing that retrieval alone does not solve version compliance.","Self-debugging against visible tests can raise visible success rates substantially, but it widens the visible-hidden gap for every model, so visible-test feedback can mask version errors that only hidden tests catch.","Larger model variants consistently outperform their smaller siblings, indicating that version-specific API recall scales with model capacity.","Coding assistants that only receive starter code without the full problem statement perform far worse, so the way version constraints are presented is itself a large factor in success."],"supporting_citations":[{"why":"GitChameleon, the predecessor benchmark, supplies the version-conditioned-generation task framing and the 116-problem baseline that GitChameleon 2.0 extends.","marker":"(Islah et al., 2024)"},{"why":"Provides the self-debugging protocol used to measure how visible-test feedback changes hidden-test success.","marker":"(Chen et al., 2023)"},{"why":"Provides the RAG method used to build documentation-augmented baselines.","marker":"(Lewis et al., 2020)"},{"why":"PyMigBench is the migration-oriented baseline the paper contrasts with static-version generation.","marker":"(Islam et al., 2023)"},{"why":"CodeUpdateArena is the synthetic API-update benchmark that motivates using real documented breaking changes instead.","marker":"(Liu et al., 2025)"},{"why":"Establishes the deprecated-API failure mode in LLM code completion that GitChameleon 2.0 quantifies under explicit version constraints.","marker":"(Wang et al., 2024b)"}],"fun_headline_variants":["Version-locked Python tasks stump top AI coders at 48-51%","Even with docs, AI solves only 59% of version-pinned coding tasks","GitChameleon 2.0: AI fails 40%+ on version-specific Python code","Top models score under 51% on library-version-aware code tests","New benchmark reveals AI's blind spot: library version mismatches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measuring instrument for every headline result is a set of hidden tests written by an AI coding agent, not independently audited by human experts; if those tests are too lenient, too strict, or encode the generator's own API mistakes, the reported 48-51% rates are not measurements of version compliance.","fun_headline_variants_meta":{"raw":{"variants":["Version-locked Python tasks stump top AI coders at 48-51%","Even with docs, AI solves only 59% of version-pinned coding tasks","GitChameleon 2.0: AI fails 40%+ on version-specific Python code","Top models score under 51% on library-version-aware code tests","New benchmark reveals AI's blind spot: library version mismatches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1389,"prompt_tokens":950,"completion_tokens":439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":566,"tokens_out":439,"duration_ms":4750,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:47:19.083539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human experts write independent oracle tests for a random sample of the 328 problems, rerun the same models against those tests, and compare pass rates with the reported hidden-test rates; a material divergence would show the benchmark's difficulty depends on the AI-generated test suite. Then classify each wrong solution as either a version-wrong API call or a logic error under the correct version; if most failures are logic errors, the paper's knowledge-retention claim would not hold.","supporting_citations":[],"review_version":1}