{"id":"352ba791-1223-4608-8098-005a299480d8","arxiv_id":"2412.04478","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark, LibEvolutionEval, measures code-completion models across library versions and finds performance varies with version, while version-aware retrieval only partially closes the gap.","lead":"This paper introduces a benchmark that tests how well code-writing AI models handle different versions of popular programming libraries like PyTorch and Matplotlib. Model accuracy changes as libraries update their functions, and version-specific documentation helps but does not fully solve the problem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-derived RAG queries make the mitigation claim uninterpretable: the Table 3 gains may reflect leakage of the target completion, not retrieval of useful documentation.","rationale":"I took the reader's weakest assumption to be data contamination, and I agree that is a real, acknowledged limitation. But the most load-bearing flaw for the paper's contribution is the oracle-conditioned retrieval query. The abstract's second sentence claims version-specific documentation improves completion; the only experiments supporting that are the RAG rows, and the query construction invalidates them as a realistic mitigation test. This concern is also more decisive than contamination: even if no GitHub snippet were memorized, the RAG comparison would still be uninterpretable because the query already encodes the answer. The reader's verdict of CONDITIONAL is the right one; the paper should be revised to rerun RAG under non-leaky queries and to release artifacts, but the version-sensitivity findings from the controlled deprecation/introduction analyses are not overturned by this concern.","tokens_in":13234,"tokens_out":5932,"duration_ms":63345,"concrete_test":"Re-run the '+Version-Aware RAG' condition with queries generated from the left context only. Use two variants: (1) the raw code prefix as the embedding query, and (2) a prompt that asks Claude or another LLM to summarize the developer's intent from the prefix alone, with the target completion withheld. Recompute the Table 3 PyTorch and Matplotlib F1 rows for all three models. If the gain over the In-File baseline collapses to within a few tenths of an F1 point (or reverses), the claimed benefit of version-specific documentation is an artifact of answer-derived queries; if the gain persists, the mitigation claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The second half of the central claim rests on the '+Version-Aware RAG' rows of Table 3. Section 2.3 describes building the retrieval query by feeding the target code completion to Claude v2 and asking it to produce a natural-language instruction without revealing the API name or arguments. That makes the query oracle-conditioned: it is generated from the exact expression the model is supposed to predict. At inference time a developer only has the prefix, not the reference, so the experimental setup leaks answer information into the retrieval step. The comparison between 'In-File' and '+Version-Aware RAG' therefore conflates the effect of version-specific documentation with a gold-answer signal; reported gains such as +4.5/+5.7 F1 for StarCoder2 and +5.4/+14.2 for GPT-4o-mini are not evidence that a deployable retrieval-augmented workflow helps. The first part of the claim (version-dependent performance) is supported by other analyses, including the CodeGen knowledge-cutoff and deprecated-API results, so it is not the load-bearing problem; the RAG mitigation claim is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LibEvolutionEval, a benchmark for version-specific code completion covering eight Python libraries across multiple versions, with two evaluation scenarios: realistic GitHub-derived code and controlled documentation-derived templates for PyTorch and Matplotlib. It evaluates several code LLMs (Mistral, StarCoder2, GPT-4o-mini, CodeGen) under in-file, version-aware, and version-aware-RAG contexts, and embedding models on version-specific documentation retrieval. The main empirical claims are that model performance varies with library version, that version-specific retrieved documentation improves F1 but does not eliminate version bias, that indirect API completions are easier than direct ones, and that deprecated and newly introduced APIs induce drops (including knowledge-cutoff effects). The paper concludes that code assistants are version-sensitive and that retrieval partially mitigates the issue.","tokens_in":13480,"tokens_out":8053,"duration_ms":74873,"significance":"LibEvolutionEval fills a real gap: existing code-completion benchmarks mostly ignore library version evolution, while this benchmark explicitly ties code-completion examples to library versions and labels APIs by introduced/deprecated/modified status. The controlled synthetic setting and the knowledge-cutoff analysis (Table 4b) are particularly valuable, as they offer a falsifiable check of model temporal awareness. If the methodological issues are fixed, the benchmark would be a solid resource for studying version sensitivity of code models and for evaluating retrieval-augmented completion. The study also provides a useful negative result: even with version-aware context, large performance variance across versions remains. The paper is not suitable for acceptance in its current form because the RAG mitigation results are uninterpretable due to query construction leakage.","major_comments":[{"comment":"The retrieval query used in the '+Version-Aware RAG' condition is generated by giving the target code completion to Claude v2 and asking it to produce a natural-language instruction without revealing the API name or arguments. This makes the query oracle-conditioned: at inference time a developer would only have the prefix, not the gold completion. Consequently, the Table 3 gains (e.g., +4.5/+5.7 F1 for StarCoder2 and +5.4/+14.2 for GPT-4o-mini) conflate the effect of version-specific documentation with a leakage of the answer into the retrieval step. The second half of the central claim, that providing version-specific documentation improves completion, is therefore not supported by the reported experiments. Please rerun the RAG condition with queries generated from the left context only (e.g., from the code prefix), or at least report an ablation in which the query is derived from the prefix and show that the gains persist.","section":"Section 2.3, Appendix C"},{"comment":"The paper acknowledges that CodeLMs may have encountered the evaluation data during pretraining, but it does not measure the degree of overlap. Because the main version-sensitivity results in the realistic scenario (Figure 5 and Table 3) are computed on GitHub-derived snippets, the observed version-dependent F1 gaps could be driven by the distribution of those snippets in the training corpus rather than by the model's inability to adapt to API evolution. Please quantify contamination (e.g., n-gram overlap with StarCoder/Stack, or report results for the controlled documentation-derived examples separately and show that the version-sensitivity pattern persists) and discuss how the conclusions change under deduplication.","section":"Appendix A"},{"comment":"The eight-library overview includes very small per-version samples: pyyaml has 8 examples per year, tqdm 12, and pillow 40. The version-by-version F1 fluctuations reported in Figure 5 for these libraries are therefore likely to be dominated by sampling noise, yet the text states that 'developer experience can vary significantly across all models and libraries.' Please report confidence intervals or otherwise restrict the breadth claim to libraries with adequate per-version sample sizes, or pool the small libraries.","section":"Section 4, Table 5"}],"minor_comments":[{"comment":"The phrase 'as they evolve over the year' is ambiguous; the data cover versions from 2019 to 2024. Use 'over multiple years' for consistency.","section":"Abstract, Section 1"},{"comment":"The manuscript does not state where the benchmark and preprocessing code will be released. Add a data and code availability statement.","section":"General"},{"comment":"The column header 'Deprecated/Intro-duced' is a line-break artifact; please use 'Deprecated/Introduced' in the table.","section":"Table 4c"},{"comment":"The three models employ different completion strategies (FIM, left-context-only, instruction-based), so absolute F1 scores across models are not directly comparable. State explicitly that only within-model comparisons (In-File vs Version-Aware vs RAG) are meaningful.","section":"Section 3, Models"},{"comment":"The subfigures (a)-(f) are referenced in an order that does not match their caption order (e.g., Figure 6b is referenced before Figure 6a). Please renumber or reference the subfigures consistently.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The reviewer's central concern, the oracle-conditioned RAG query, is valid and fixable within the scope of a major revision. The benchmark itself is timely and within the scope of cs.SE. I do not see a fundamental novelty problem, but the paper needs a dedicated contamination analysis and a re-run of the RAG experiments with prefix-derived queries before the mitigation claim can be evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThis paper is worth your time because it tackles a real, under-served problem: code completion when libraries change. The contribution is a benchmark, LibEvolutionEval, with version-specific examples from eight libraries, built two ways: realistic snippets from permissively licensed GitHub repos (with version matching via requirements.txt or PyPI dependency matrices) and synthetic templates generated from API documentation. The version labeling and the granularity classification into introduced/deprecated/modified/unchanged APIs are useful. The strongest empirical finding is that CodeGen's performance drops sharply on APIs introduced after its 2022 cutoff, which gives a clean method for estimating a model's knowledge cutoff. The converse, newer models failing on deprecated APIs, is also shown. These analyses are independent of the benchmark's main RAG claim, and they hold up.\n\nThe weakness is the RAG mitigation experiment in Table 3. The retrieval query is generated by feeding the target completion to Claude and asking for a natural-language instruction without revealing the API name or arguments. That is oracle-conditioned. A developer at inference time has the prefix, not the reference completion. So the '+Version-Aware RAG' rows are inflated by leakage from the answer, and the reported F1 gains do not demonstrate that version-aware retrieval works in practice. This is a serious flaw, not a minor one, because the abstract's second claim rests on it. The first claim, that library evolution affects model performance, is supported by the deprecated/introduced analyses and the controlled scenario, so it survives.\n\nMinor issues: no artifact link in the paper, no error bars, and contamination is acknowledged but not measured. The controlled scenario mitigates memorization concerns for the synthetic data, but the realistic numbers could still be partially memorization artifacts.\n\nI'd send this to a serious referee. The benchmark is novel and the knowledge-cutoff result is publishable. The RAG experiment needs to be redone with prefix-only queries before the mitigation claim can be believed. This is a conditional accept, not a reject.\n\nBest.","headline":"Useful benchmark for version-sensitive code completion, but the RAG mitigation claim is undermined by oracle-derived retrieval queries.","tokens_in":13986,"tokens_out":3631,"would_cite":true,"duration_ms":36537,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Public library evolution measurably degrades code completion models.","keywords":["code completion","library evolution","API deprecation","version-specific documentation","retrieval-augmented code generation","benchmark","embedding models","knowledge cutoff"],"falsifier":"Run the same version-specific completions on snippets that have been paraphrased or had identifiers renamed so they no longer match any training example, then compare F1; if the version-dependent gaps disappear under paraphrase, the measured evolution bias is largely memorization, while persistence would support the paper's interpretation.","tokens_in":13061,"feed_emoji":"📦","tokens_out":3961,"duration_ms":37623,"temperature":0.7,"pith_summary":"LibEvolutionEval tests whether code completion models can keep up as public libraries change their APIs. The paper argues that model performance tracks library versions: scores fluctuate as torch, matplotlib, and six other libraries release new versions, and models systematically struggle with deprecated APIs. Feeding models retrieved documentation for the exact library version improves completion F1, but a version-dependent bias remains. The result matters because real developers work across old and new library versions, so a coding assistant that is right for one version can be wrong for another.","feed_headline":"Library evolution measurably degrades code completion models","feed_subtitle":"Version-specific API docs help recover accuracy, but a new benchmark shows models keep a built-in version bias.","key_machinery":"The load-bearing object is the version-specific code-completion benchmark itself: a dataset of in-line completions, each anchored to a documented library version and annotated by API relationship (direct versus indirect) and by granularity (introduced, deprecated, modified, unchanged). Direct completions are import-prefixed calls such as torch.nn.ReLU(); indirect completions resolve a method on an object created by an earlier API call. The paper also builds version-stamped documentation databases for PyTorch and Matplotlib and uses an embedding model to retrieve the top three matching API docs, which are injected as commented context. The F1 score over extracted API expressions carries the evaluation, and Mean Reciprocal Rank carries the retrieval analysis.","core_discovery":"The paper's central claim is that public library evolution, not just code context, drives code completion accuracy. Across eight libraries and several open models, the authors show F1 varies substantially from version to version, with deprecated APIs consistently harder than the overall set and newly introduced APIs exposing a model's training cutoff. They introduce a benchmark with two complementary settings---realistic snippets from permissively licensed repositories and controlled templates built from API documentation---and show that adding retrieved version-specific documentation improves performance (for example, StarCoder2-7B rises from 68.8 to 73.3 F1 on PyTorch) without eliminating the underlying version bias. The conclusion is that developers using code assistants with different library versions experience different accuracy, and fixes like retrieval are partial.","pith_inferences":["The benchmark could be reused as a contamination test: if a model scores high on realistic GitHub examples but low on paraphrased or identifier-renamed versions of the same API calls, memorization rather than version understanding is at play.","Version-stamped documentation retrieval mitigates surface errors but will not fix internal version priors, suggesting that explicit version tokens in prompts or calibrated confidence scores are worth exploring.","The direct-versus-indirect gap hints that models track object provenance poorly, so a testable extension is whether chain-of-thought reasoning about the originating API improves indirect completions.","Library-evolution benchmarks could be extended beyond Python to other ecosystems to measure how widely version bias affects assistant reliability and legacy-code maintenance."],"forward_implications":["Code completion benchmarks that ignore library version overstate or hide model capability, so version-controlled evaluation is needed.","Version-aware retrieval should become a standard component in coding assistants, since it consistently improves completion F1.","Deprecated-API performance can serve as a probe of a model's temporal knowledge; for instance, CodeGen-1.0's drop on 2023-2024 introduced APIs aligns with its 2022 knowledge cutoff.","Scaling models improves overall performance but does not remove version-to-version fluctuations, pointing to a need for versioned fine-tuning or explicit temporal training signals.","Embedding models used for retrieval also drift with library versions, which explains why retrieval augmentation cannot fully close the version gap."],"supporting_citations":[{"why":"Supplies the F1-based API-completion evaluation methodology and cross-file completion benchmark that LibEvolutionEval adapts.","marker":"(Ding et al., 2023b)"},{"why":"Provides the retrieve-and-generate repository-level completion approach adapted for version-specific documentation retrieval.","marker":"(Zhang et al., 2023)"},{"why":"CodeSage is the default embedding model used to retrieve version-specific API documentation.","marker":"(Zhang et al., 2024)"},{"why":"StarCoder2 is one of the primary code completion model families evaluated across library versions.","marker":"(Lozhkov et al., 2024)"},{"why":"CodeGen 1.0 is used to demonstrate knowledge-cutoff effects through its decline on newly introduced APIs.","marker":"(Nijkamp et al., 2023)"},{"why":"Gorilla establishes prior evidence that zero-shot API code completion suffers from hallucination, motivating the benchmark.","marker":"(Patil et al., 2023)"},{"why":"Provides the empirical framing of API stability and adoption that motivates studying library evolution.","marker":"(McDonnell et al., 2013)"}],"fun_headline_variants":["Library evolution throws off code completion models","Version-specific docs don't fix code AI's library bias","Code completion accuracy shifts with library versions","New benchmark reveals code models' version blindspots","Library updates expose training gaps in code AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the benchmark's version-sensitive scores actually measuring API-evolution understanding, which requires that the evaluated models did not memorize the exact GitHub snippets used as evaluation examples; the paper's own limitations section concedes that code LMs are trained on vast unlabeled code repositories and may have seen this data.","fun_headline_variants_meta":{"raw":{"variants":["Library evolution throws off code completion models","Version-specific docs don't fix code AI's library bias","Code completion accuracy shifts with library versions","New benchmark reveals code models' version blindspots","Library updates expose training gaps in code AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1322,"prompt_tokens":863,"completion_tokens":459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":391}},"tokens_in":479,"tokens_out":459,"duration_ms":5468,"temperature":1.0,"reasoning_tokens":391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:05:20.428123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same version-specific completions on snippets that have been paraphrased or had identifiers renamed so they no longer match any training example, then compare F1; if the version-dependent gaps disappear under paraphrase, the measured evolution bias is largely memorization, while persistence would support the paper's interpretation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CodeSage is the default embedding model used to retrieve version-specific API documentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CodeGen 1.0 is used to demonstrate knowledge-cutoff effects through its decline on newly introduced APIs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the empirical framing of API stability and adoption that motivates studying library evolution."}],"review_version":1}