REVIEW 2 major objections 6 minor 35 references
GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Best AI coders pass only half of version-locked Python tasks.
desk verdict A genuine, reusable version-conditioned benchmark with execution-based evaluation; the AI-generated test audit is the main thing to fix, but the central result is solid. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is GitChameleon 2.0: 328 Python code-completion tasks, each tied to a documented breaking change in one of 26 libraries, with the library and exact version named in the prompt and installed in the validation environment. The key design move is version-conditioned generation (VCG), which forces models to write code for a fixed, in-distribution version rather than migrate code forward; the hidden-test suite is the scoring instrument, with visible tests reserved for self-debugging and reference documents reserved for RAG experiments. A supporting mechanism is the logic-vs-knowledge decomposition, which counts logic-related AST nodes in ground-truth solutions and uses the low counts to argue that the benchmark measures knowledge retention, not complex code logic.
What would settle it
Have human experts write independent oracle tests for a random sample of the 328 problems, rerun the same models against those tests, and compare pass rates with the reported hidden-test rates; a material divergence would show the benchmark's difficulty depends on the AI-generated test suite. Then classify each wrong solution as either a version-wrong API call or a logic error under the correct version; if most failures are logic errors, the paper's knowledge-retention claim would not hold.
Extended reading notes
Core claim
The paper's central claim is that state-of-the-art LLM-based coding systems are unreliable when asked to write code for a library version they have already seen. Even with the target version stated in the prompt and installed in the test environment, the best enterprise model reaches 51.2% hidden-test success and several large models cluster at 48-51%; the best retrieval-augmented result is about 59%, so more than 40% of problems remain unsolved even with documentation available. The benchmark deliberately uses versions inside training windows and mostly short solutions with few logic nodes, so it interprets the failures as a version-knowledge disambiguation problem rather than a code-reasoning problem. The paper's conclusion is that version-conditioned code generation is a measurable, currently unsolved subproblem of code generation, and that execution-based benchmarks are needed to see it.
Load-bearing premise
The measuring instrument for every headline result is a set of hidden tests written by an AI coding agent, not independently audited by human experts; if those tests are too lenient, too strict, or encode the generator's own API mistakes, the reported 48-51% rates are not measurements of version compliance.
Editorial extensions
If this is right
- Teams that ask an LLM to produce code for a pinned library version can expect roughly one in two completions to fail hidden functional tests, so version-aware guardrails and testing are necessary before such output can be shipped.
- Providing relevant documentation through RAG lifts the best model to about 59% hidden-test success, yet leaves over 40% of problems unsolved, showing that retrieval alone does not solve version compliance.
- Self-debugging against visible tests can raise visible success rates substantially, but it widens the visible-hidden gap for every model, so visible-test feedback can mask version errors that only hidden tests catch.
- Larger model variants consistently outperform their smaller siblings, indicating that version-specific API recall scales with model capacity.
- Coding assistants that only receive starter code without the full problem statement perform far worse, so the way version constraints are presented is itself a large factor in success.
Reading between the lines
- Editorial extension: because the benchmark pins versions inside training windows, the 48-51% ceiling suggests pretraining stores versioned API knowledge in a poorly disentangled way; a targeted error analysis distinguishing stale-API calls from hallucinated newer APIs would make this precise.
- Editorial extension: the hidden tests were generated by an AI agent and not independently human-audited, so auditing a random subset with human-written oracle tests would show how much of the reported difficulty is the test suite itself.
- Editorial extension: since most ground-truth solutions have few logic nodes, an oracle-document RAG condition would isolate whether remaining failures come from retrieval inaccuracy or from models ignoring correctly retrieved version-specific documentation.
- Editorial extension: the paper leaves out human baselines and version-to-version translation; adding a human-performer baseline would turn the absolute 48-51% numbers into a calibrated statement about how hard version-conditioned generation is for humans versus models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. GitChameleon 2.0 introduces a dataset of 328 Python code-completion problems, each tied to a specific library version and a documented breaking change, with executable hidden tests and a visible test per problem. The paper evaluates a broad range of open-weight and enterprise LLMs, tool-calling agents, CLI/IDE coding assistants, and RAG pipelines under greedy decoding, chain-of-thought, self-debugging, multi-step agent, and retrieval-augmented settings. The central finding is that enterprise models achieve only 48-51% hidden-test success under greedy decoding, with self-debugging and RAG providing gains yet leaving over 40% of problems unsolved. The paper also reports analyses by API change category, error type, and version release year, and argues that the benchmark primarily measures version-specific knowledge retention rather than complex logic.
Significance. If the hidden tests are valid, this is a useful benchmark and empirical study. Strengths include execution-based evaluation in Docker containers with pinned library versions, standard errors on all headline numbers, a wide model/agent/assistant coverage, public dataset and code, and the visible/hidden test gap analysis that supports the self-debugging findings. The moderate correlations with SWE-bench and LiveCodeBench are thought-provoking. The main weakness is that the hidden tests were AI-generated without independent verification, and the knowledge-retention interpretation rests on a subjective AST classification; these need to be addressed before the headline claims can be fully trusted.
major comments (2)
- [Appendix A.4 (Tables 1-4)] The benchmark's headline metric is hidden-test success, yet the hidden tests were generated by the Zencoder agent (built on GPT-4.1) with no independent human audit; the reported 96.5% coverage and the fact that reference solutions pass do not establish that the tests correctly discriminate version-correct from version-incorrect solutions. Coverage measures how much of the reference implementation is exercised, not whether the assertions encode the version-specific behavior described in each problem statement, and a reference solution passing its own tests is necessary but not sufficient for test validity. Because Zencoder is itself an LLM, the tests can encode the same API misconceptions the benchmark aims to detect (e.g., using a deprecated argument in the reference implementation or failing to check return types that changed across versions). I request an independent human audit of a sample of hidden tests, or at least an adversarial validation in which known version-incorrect solutions (e.g., using the pre-change API) are shown to fail, and a report of the resulting precision/recall of the test suite. Without this, the 48-51% enterprise rates in Table 1 and all downstream comparisons rest on an unvalidated measuring instrument.
- [Appendix H (Table 15)] The claim that GitChameleon 2.0 'is primarily designed to assess version-specific knowledge retention rather than complex logic-based code generation' rests on an ad-hoc AST node classification whose categories are not independently validated. Table 15 counts 'calling a math or utility function with non-obvious purpose' and 'composing multiple calls together' as logic-related, yet these categories require judgment; no inter-annotator agreement or comparison with a validated complexity measure (e.g., cyclomatic complexity or human ratings) is provided. Since this claim is used to interpret the benchmark's results and to position the contribution relative to logic-reasoning benchmarks, it should be supported by a more rigorous validation, or the wording should be softened to describe the observed distribution of logic nodes without asserting the benchmark's primary design intent.
minor comments (6)
- [Figure 1] The function name 'custom_violinpolot' in the figure appears to be a typo; it should probably be 'custom_violinplot'.
- [Appendix F.2] The code listing shows 'from scipy import llinalg' with a double 'l'; this is likely a typo for 'linalg'.
- [Table 3] Cline with GPT-4.1 and Cline with GPT-4.1-nano report identical hidden success rates (54.6%) and identical API hit rates (48.8%); please verify whether this is a copy-paste error or an actual duplicate result.
- [Appendix J.1] The Kymatio entry lists a librosa documentation URL instead of the Kymatio project URL; please correct this citation.
- [Appendix A.4] Capitalization of 'Zencoder' is inconsistent; the text uses both 'Zencoder' and 'ZENCODER'.
- [Table 14] The entry 'LLM-Deprecated-APl' contains a typo; 'APl' should be 'API'.
Circularity Check
No load-bearing circularity: results are execution-based against pinned versions; only a minor self-citation and a mild overlap between the hidden-test generator and an evaluated model.
full rationale
The paper's core claims—enterprise models reaching 48-51% hidden-test success, self-debugging/RAG gains, and the version-conditioning difficulty—are produced by executing candidate solutions in pinned Docker environments against hidden unit tests, an external measurement rather than a derivation from the benchmark's own assumptions. Appendix A.4 describes hidden tests generated by Zencoder (built on GPT-4.1) with execution feedback and manual patching; although GPT-4.1 is also an evaluated model, the pass/fail labels are grounded in real execution against installed library versions, and no fitted parameter or predicted quantity is defined in terms of those labels. The only self-citation is the predecessor GitChameleon (Islah et al., 2024), mentioned as context in Section 4; it does not support any headline number. Appendix H's claim that the benchmark targets knowledge retention rather than logic is a transparent design statement based on an explicit AST taxonomy (Table 15) that excludes library-method calls from 'logic-related' nodes; this is definitional by design but does not feed back into the experimental results. No step reduces to Equation X = Equation Y by construction or renames a fit as a prediction, so the score is 2 for the minor self-citation and the noted test-generation overlap, not for a circular derivation.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper The versions used in the benchmark fall within the training data of the evaluated models.
- domain assumption The hidden tests correctly encode the version-specific behavior of each problem.
- domain assumption Changelogs and release notes for the selected libraries accurately document the breaking changes.
- ad hoc to paper The AST logic-node count is a valid proxy for distinguishing knowledge retention from logic reasoning.
Cite this review
Pith. "Pith review of GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities." pith.science (2026). https://pith.science/paper/A4PTODPV
@misc{pith2026250712367,
author = {Pith},
title = {Pith review of: GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities},
year = {2026},
howpublished = {\url{https://pith.science/paper/A4PTODPV}},
note = {Machine review of arXiv:2507.12367}
}
read the original abstract
The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent version updates while preserving backward compatibility. While existing code evolution benchmarks provide valuable insights, they typically lack execution-based evaluation for generating code compliant with specific library versions. To address this, we introduce GitChameleon 2.0, a novel, meticulously curated dataset comprising 328 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. GitChameleon 2.0 rigorously evaluates the capacity of contemporary large language models (LLMs), LLM-powered agents, code assistants, and RAG systems to perform version-conditioned code generation that demonstrates functional accuracy through execution. Our extensive evaluations indicate that state-of-the-art systems encounter significant challenges with this task; enterprise models achieving baseline success rates in the 48-51% range, underscoring the intricacy of the problem. By offering an execution-based benchmark emphasizing the dynamic nature of code libraries, GitChameleon 2.0 enables a clearer understanding of this challenge and helps guide the development of more adaptable and dependable AI code generation methods. We make the dataset and evaluation code publicly available at https://github.com/mrcabbage972/GitChameleonBenchmark.
Figures
Figures from the paper (18 more)
Reference graph
Works this paper leans on
-
[1]
Sample ID: 0, Logic Nodes: 3 import torch def log _ ndtr ( input _ tensor : torch . Tensor ) -> torch . Tensor : import numpy as np from scipy . stats import norm output = torch . from _ numpy ( norm . logcdf ( input _ tensor . numpy () ) ) return output Listing 37: Sample 0 Ground Truth Solution
-
[2]
Sample ID: 329, Logic Nodes: 0 import matplotlib . pyplot as plt def use _ seaborn () -> None : plt . style . use ( " seaborn " ) Listing 38: Sample 329 Ground Truth Solution 0 20 40 60 80 100 Number of Logic Nodes 0 50 100 150 200 250Frequency Logic Nodes Distribution Figure 14: Logic Nodes Distribution over samples’ ground truth solutions’ ASTs. Most gr...
work page 2019
-
[5]
Prompt alchemy: Automatic prompt re- finement for enhancing code generation. Preprint, arXiv:2503.11085. Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. DocPrompting: Generating code by retrieving the docs. 13 A Benchmark Details This appendix provides additional details on the GitChameleon 2.0 benchmark. We provid...
arXiv 2024
-
[6]
Output only the code block and nothing else . Example output format : ``` python # [ Your code here , incorporating the starter code ] # [ Additional code and comments as needed ] ``` After writing your solution , please review it to ensure all requirements are met and the code is correct and efficient . Here are the key elements for this task : (b) Syste...
-
[7]
Avoid interactive , stateful , or environment - dependent constructs ( e . g . , Django projects , web servers )
-
[18]
Ensure your code is clean , efficient , and well - commented
-
[19]
Output nothing else after the code block . Example output format : [ Step - by - step thinking ] ``` python # [ Your code here , incorporating the starter code ] # [ Additional code and comments as needed ] ``` After writing your solution , please review it to ensure all requirements are met and the code is correct and efficient . Here are the key element...
-
[20]
Required Library : < library > {{ library }} </ library >
Show all 35 references
-
[21]
Python version : < python > {{ python_version }} </ python >
-
[22]
Coding Problem : < coding_problem > {{ coding_problem }} </ coding_problem >
-
[23]
Your task is to fix issues in a generated Python solution for a given programming problem
Starter Code : < starter_code > {{ starter_code }} </ starter_code > 35 • Qwen 2.5-VL Instruct 72B : (Qwen et al., 2025) • Qwen 3 235B :(Yang et al., 2025) • Command A 111B : (Cohere et al., 2025) • DeepSeek R1 685B : (DeepSeek-AI, 2025) • DeepSeek v3 : (DeepSeek-AI et al., 20...
2025
-
[24]
Use the required libraries explicitly in your code
-
[25]
Correctly incorporate the provided starter code - do not remove or alter its structure
-
[26]
Write in standard Python syntax
-
[27]
Wrap your entire solution within a single Markdown code block
-
[28]
Do not include any text outside the code block - no explanations , comments , docstrings , or usage examples
-
[29]
Ensure the code is clean , efficient , and syntactically valid
-
[31]
Your output must be executable in a non - interactive environment ( e . g . , a test harness or script runner ) . Example output format : ``` python # [ Your corrected code here ] ``` Before submitting , carefully review your code for correctness , completeness , and adherence...
-
[32]
Use only the specified libraries and respect the given version constraints
-
[33]
Incorporate any provided starter code as required
-
[34]
Do not provide anything in the response but the code
Write only Python code - no in - line comments or usage examples . Do not provide anything in the response but the code
-
[35]
Ensure the code is clean , minimal , and adheres to best practices
-
[36]
{ assistant name }
The code must be executable in a non - interactive environment ( e . g . , avoid frameworks like Django or code requiring a web server ) . Context : { context } Based on the above , respond to the user query below . Query : { query } Here, {context} refers to the context of th...
-
[37]
Give a formalized explanation of the keyword using technical languages . Provided Format : Keywords :[ Keywords ] Explainations :[ Formalized explanations ] Guidelines : - Prioritize keywords that are crucial to understanding the input parameters , return content or supplement...
-
[38]
Use the required library in your solution
-
[39]
Incorporate the provided starter code correctly
-
[40]
Write your solution in Python
-
[41]
Format your solution within a markdown code block
-
[42]
Ensure your code is clean and efficient
-
[43]
Do not add any in - line comments , documentations , references or usage examples
Output only the code block and nothing else . Do not add any in - line comments , documentations , references or usage examples
-
[44]
For example , do not write code which requires building a Django project or deploying a web - app
Make sure your code is executable in a non - interactive environment . For example , do not write code which requires building a Django project or deploying a web - app . Example output format : ``` python # [ Your code here , incorporating the starter code ] ``` After writing...
-
[2008]
In Proceedings of the 7th Python in Science Conference, pages 11–15
Exploring network structure, dynamics, and function using NetworkX. In Proceedings of the 7th Python in Science Conference, pages 11–15. Charles R Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cour- napeau, Eric Wieser, Julian Taylor, Seb...
2020 arXiv
-
[2021]
Evaluating large language models trained on code. ArXiv. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Al...
2023 arXiv
-
[2024]
Preprint, arXiv:2410.15966
Self-explained keywords empower large language models for code generation. Preprint, arXiv:2410.15966. Google. 2025. Grounding with Google Search | Gemini API. https://ai.google.dev/gemini-api/doc s/grounding. Aric A Hagberg, Daniel A Schult, and Pieter J Swart
2025 arXiv
-
[2025]
https://github.com/huggingfa ce/smolagents
‘smolagents‘: a smol library to build great agentic systems. https://github.com/huggingfa ce/smolagents. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Ta- laei Khoei. 2025. Agentic retrieval-augmented generation: A survey on agentic rag. Preprint, arXiv:2501.09136. Jianlin...
2025 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.