{"id":"39f9da92-9b9c-444d-a1c1-223a8dc6dc46","arxiv_id":"2502.00519","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CoDocBench provides 4,573 coupled code-docstring changes from GitHub and reports that Llama-3.1 and Mixtral have limited success at aligning and updating code-docstring pairs.","lead":"This paper introduces CoDocBench, a dataset of 4,573 paired code and docstring changes mined from real GitHub commits. It also shows that two large language models struggle to keep code documentation aligned after code changes, suggesting a useful benchmark for software maintenance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The curation pipeline validates syntactic co-change but never checks semantic coupling between docstring and code edits; without this check, the reported model difficulty could be partly an artifact of label noise.","rationale":"The paper's primary contribution is the dataset, and its usefulness depends on the extracted old/new code-docstring pairs being genuinely aligned. The pipeline mechanically checks co-change but not semantic relatedness; Tree-Sitter validates syntax, not whether the docstring edit documents the code edit. The reader's weakest assumption was that the regex/AST pipeline captures genuine coupled changes without systematic bias, and specifically flagged missed renames/moves. I agree that rename/move insensitivity is a real coverage bias, but the more broadly load-bearing issue is that every included sample may contain unrelated co-changes, making the ground-truth target ill-posed for the alignment tasks. The public dataset and code make the proposed annotation test feasible, so the concern is checkable rather than speculative. Because the authors already received a conditional verdict and this concern reinforces the need for validation, the reader's verdict does not change; the conditional status should include a requirement to demonstrate semantic coupling or to qualify the difficulty claim.","tokens_in":6673,"tokens_out":14587,"duration_ms":153221,"concrete_test":"Uniformly sample 100 test entries; for each, present annotators with old code, new code, old docstring, and new docstring (masking commit messages and file paths) and ask whether the docstring change is semantically related to the code change. Compute inter-annotator agreement (Cohen's kappa) and the fraction judged related. If the relatedness fraction is below about 70% or agreement is poor, the dataset has not demonstrated genuine coupling, and the difficulty claim should be re-evaluated after filtering to semantically related samples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II's post-processing filters samples to those where both the code and docstring changed and excludes whitespace-only edits, then uses Tree-Sitter and function_parser to validate parsing; it never validates that the docstring diff is semantically related to the code diff. The RQ1/RQ2 tasks in Section III treat the repository's new docstring as the ground-truth aligned target for the new code. If a substantial share of the 4573 samples are commits where the docstring edit is an independent typo fix, style rewrite, or documentation-only update, then the old/new docstring pair is not the aligned target the prompts ask models to produce, and the low accuracies in Tables II and III would reflect noisy labels rather than genuine task difficulty. This affects every included sample, whereas the Section V limitation about missed renames/moves to differently named files removes only a subset of commits. The central claim that CoDocBench is a natural dataset of coupled code-documentation changes therefore rests on an unvalidated semantic-coupling assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoDocBench, a dataset of 4,573 Python code-docstring co-change samples mined from popular GitHub projects. The construction pipeline uses PyDriller over consecutive commits of same-named files on the main branch, detects function and docstring changes with regular expressions, filters to instances where both code and docstring changed (excluding whitespace-only edits), and validates parsing with Tree-Sitter and function_parser. The paper defines two tasks: RQ1 (temporal alignment of generated code/docstrings with old/new references) and RQ2 (generating an updated docstring from new code, or updated code from a new docstring). It evaluates Llama-3.1 405B Instruct Turbo and Mixtral 8x22B Instruct v0.1 under zero-shot, contextual-information, BM25 3-shot, and hybrid prompting, reporting raw counts of 'correct' alignments in Tables II and III. The authors conclude that current models find these maintenance-related tasks challenging. The paper also includes limitation statements about same-file tracking and main-branch-only mining.","tokens_in":6844,"tokens_out":4336,"duration_ms":45334,"significance":"CoDocBench addresses a real gap: there is no widely used, large benchmark specifically for coupled code-docstring evolution in software maintenance. If the dataset genuinely contains semantically coupled code and docstring changes, it provides a useful resource for training and evaluating models on documentation-update tasks, with a public release and concrete evaluation tasks. However, the significance is currently bounded by two unsupported assumptions: that syntactic co-change implies semantic coupling, and that raw edit-distance alignment counts are a valid measure of task success. If these are addressed, the dataset could be a valuable contribution to the software engineering and ML4Code communities.","major_comments":[{"comment":"The pipeline establishes only syntactic co-change: both the code and the docstring changed in the same commit, and both parse correctly. It never validates that the docstring edit is semantically related to the code edit. The abstract and Section I define the dataset as containing 'coupled changes', but a commit with an unrelated docstring typo fix alongside a code refactor would pass the current filters. This matters for every sample in the 4,573-sample dataset and for the difficulty claims in Tables II and III, because the repository's new docstring is treated as the ground-truth aligned target for the new code. Please add a validation step, such as manual annotation of a random sample or a quantitative semantic-agreement check between the code diff and the docstring diff, and report the resulting precision. Without this, the central claim of coupling is unsupported.","section":"Section II (Dataset Collection Methodology)"},{"comment":"For RQ1, a correct response requires both the old-generation-to-old-reference comparison and the new-generation-to-new-reference comparison to succeed. If each binary comparison is at chance (p=0.5), the expected accuracy for 'Aligned: Both' is 25%. The reported best count is 407/2273 = 17.9% for Llama-3.1 405B and 352/2273 = 15.5% for Mixtral, both below that chance level. This indicates that the models are not merely struggling but are likely systematically biased (for example, generating a single temporal style regardless of input), and the task should be described relative to this chance baseline rather than as an unqualified demonstration of difficulty. Please report chance baselines, per-condition percentages, and confidence intervals or significance tests for all counts in Tables II and III.","section":"Section IV, Table II (RQ1 results)"},{"comment":"The 'correct' criterion in RQ2 is based entirely on raw edit distance, with no validation that edit distance agrees with human judgments of whether the generated update matches the intended change. Edit distance is sensitive to surface form and tokenization, and the paper's own statistics in Figures 3-5 show that many code and docstring diffs are small, so the counts in Table III may partly reflect spurious surface similarity rather than genuine alignment with the new reference. Please validate the metric on a sample of outputs (for example, by human rating or a secondary semantic similarity measure) and report agreement; otherwise the interpretation of the raw counts is not reliable.","section":"Section IV (RQ2 evaluation definition)"},{"comment":"The 'high-quality' characterization of the source projects is asserted rather than demonstrated. The selection criterion 'top 200 Python projects on GitHub' is not operationalized: no star threshold, activity cutoff, or candidate pool is given, and star count is a popularity signal rather than a direct quality measure. The same-file tracking limitation is acknowledged, and it is important because documentation updates that accompany function moves or renames are systematically excluded, which may bias the dataset toward small, local edits. Please quantify the extent of this exclusion if possible, and temper the 'natural' and 'high-quality' claims accordingly.","section":"Section II and Section V (Limitations)"}],"minor_comments":[{"comment":"The name 'Schrek et al.' in the introduction is spelled 'Schreck et al.' in the reference list; please unify the spelling. Also, 'DocStrings' is capitalized inconsistently with 'docstring' elsewhere.","section":"Throughout"},{"comment":"The exact prompt templates for RQ1, RQ2, and the prompting strategies are not given. Because the evaluation depends on the model outputs, the prompts should be included verbatim in an appendix for reproducibility.","section":"Section III (Research Questions)"},{"comment":"Tables II and III report raw counts without the corresponding percentages or totals in the table cells. Adding percentages and including the 25% chance baseline for RQ1 would make the results much easier to interpret.","section":"Section IV (Results)"},{"comment":"Generation settings such as temperature, top-p, max tokens, and number of decoding runs are not reported. Since edit-distance comparisons can be sensitive to sampling, these details should be stated.","section":"Section IV (Models)"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about semantic coupling is valid and is the main load-bearing issue: the dataset construction validates syntactic co-change but not semantic coupling. The paper is within scope for a software engineering venue and the resource is potentially useful, but the evaluation claims need stronger statistical grounding and the metric needs validation. I recommend major revision rather than reject, because the underlying data collection is credible and the limitations are likely addressable with additional analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoDocBench is a real, checkable new asset: 4,573 samples of Python functions whose code and docstring both changed in the same commit, mined from top GitHub projects via PyDriller and validated with Tree-Sitter. That target—same-commit code+docstring co-change—is not covered by existing summarization or search datasets, and the authors release code and data on Zenodo/GitHub, so the artifact itself is reproducible. The paper is honest about the main-branch and same-filename limitation. Credit where due: the construction pipeline is described in enough detail to re-implement, and the task framing (update code given new docstring, and vice versa) is a sensible way to operationalize documentation maintenance.\n\nThe soft spots are real, though. The biggest one: 'coupled' is never checked semantically. The pipeline only verifies that both the code string and the docstring string changed and parsed; it never checks that the docstring edit is about the same change as the code edit. A commit could fix a typo in a docstring and refactor code in the same commit, and that sample enters the dataset as a 'coupled change.' That contaminates the ground truth for both RQ1 and RQ2. The authors do not report any manual or automatic sampling to estimate what fraction of the 4,573 samples are genuinely aligned. This is a load-bearing assumption and it is unvalidated. The stress-test note is right: this affects every sample, not just the moved-function cases the authors acknowledge.\n\nSecond, the evaluation is under-powered: results are raw counts with no confidence intervals, no significance tests, and no chance baseline. For RQ1, the 'correct' definition requires both old and new alignments, but the paper doesn't report how often a trivial output (e.g., copying the input) would pass. Edit distance is a coarse proxy and is not validated against human judgment. Third, the dataset is small (4,573 samples, 60% from 25 projects), so the 'large' claim is modest.\n\nWho is this for? Researchers working on docstring maintenance or LLM code editing who need a starting point and are willing to audit the data themselves. It deserves a serious referee because the resource is new and the authors are transparent, but the paper needs significant revision: add a semantic-coupling validation (manual sample, or an LLM-based filter), report statistical grounding, and calibrate chance. I'd send it to review, but with the expectation of heavy revision.","headline":"A genuinely new dataset resource for code-docstring co-change, but the 'coupled' claim is never semantically validated and the evaluation lacks statistical grounding.","tokens_in":7366,"tokens_out":2373,"would_cite":true,"duration_ms":25225,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real paired code-docstring edits expose an LLM alignment gap.","keywords":["code-documentation alignment","docstring","software maintenance","dataset construction","GitHub mining","large language models","code generation","benchmark"],"falsifier":"A random-sample audit of commits where a function is moved to a differently named file, counting how often its docstring changes in the same commit, would settle whether the dataset's same-file tracking biases its samples; re-running the evaluations with a semantic similarity metric instead of edit distance would settle whether the reported difficulty is an artifact of the metric.","tokens_in":6487,"feed_emoji":"🔗","tokens_out":4399,"duration_ms":42221,"temperature":0.7,"pith_summary":"The paper introduces CoDocBench, a dataset of 4,573 coupled changes mined from high-quality GitHub projects, where each sample is one commit in which a function's code and its docstring changed together. The authors' aim is to support two maintenance tasks: writing a docstring that matches new code, and writing code that matches a new docstring. On a 2,273-sample test split, they evaluate two large language models and report that correct temporal alignment of both old and new versions is rare, with the best model aligning both references in only 407 of 2,273 cases in the first task. This matters because documentation is known to lag behind code changes, so a natural benchmark of realistic paired edits could train and measure models that help developers keep documentation in sync.","feed_headline":"Real paired code-docstring edits expose LLM alignment gap","feed_subtitle":"Mined from 4,573 genuine commits, the benchmark shows even top models align both versions in under 18% of tries.","key_machinery":"The carrying object is the CoDocBench dataset itself: each entry holds the old and new versions of a function's code and docstring, plus commit metadata, unified diffs, and file and function identifiers. The extraction machinery is a commit-diff pipeline that mines each file's commit history, detects function definitions and triple-quoted docstrings with regular expressions, and validates the function-name, code, and docstring associations with a syntax-tree parser and a function parser. The evaluation mechanism is edit distance: a generated output counts as correctly aligned when it is closer to the reference of its own temporal side than to the other side's reference. This machinery makes the benchmark reproducible and the difficulty claim measurable.","core_discovery":"The central claim is that naturally occurring, coupled code-docstring changes can be extracted from real repositories at scale, and that this resource exposes a real capability gap in current large language models. The paper's construction pipeline takes Python files from popular, actively maintained projects, tracks each function across consecutive commits to the same file on the main branch, and keeps only entries where both the code and the docstring were edited together while excluding whitespace-only changes. Validation with a syntax-tree parser and a function parser confirms the function-docstring association. Under an edit-distance alignment criterion, the better model achieves 407 out of 2,273 correct 'both old and new' alignments for code/docstring generation, and at most 1,311 correct updates when the model is given the counterpart's new version, evidence that the tasks are feasible to pose but not yet solved.","pith_inferences":["Because the dataset only follows functions inside same-named files on the main branch, its natural distribution is likely biased toward small, local edits; moves and renames that co-occur with docstring changes are systematically absent.","The edit-distance alignment criterion rewards surface-level similarity, so a semantic or execution-based metric could change the reported difficulty ranking.","The heavy project skew, with the top 25 projects supplying about 60% of samples, means reported accuracies may not transfer to less popular or differently documented codebases.","A direct test of the dataset's utility would be fine-tuning a smaller model on the training split and measuring whether alignment accuracy improves over the zero-shot baselines reported here."],"forward_implications":["A reusable benchmark now exists for training and evaluating docstring-update and code-update models on realistic, commit-level paired edits.","The documented tendency for docstrings to lag code changes becomes a concrete target: models that pass CoDocBench-style alignment tests could help close that gap.","The asymmetric results, where docstring updates are easier than code updates and adding contextual information helps while BM25 few-shot retrieval mostly does not, give immediate design guidance for maintenance assistants.","The project-based 50/50 split makes the benchmark a tougher generalization test and supports future comparisons between models and methods.","The public release of the dataset and schema allows other researchers to add tasks such as commit-message generation or diff summarization using the same paired-change samples."],"supporting_citations":[{"why":"Supplies the motivating estimate that docstrings are updated only about one-third as often as code.","marker":"[7]"},{"why":"Provides the Git-mining framework used to iterate over commits for each Python file.","marker":"[8]"},{"why":"Provides the syntax-tree parser used to fix function-to-docstring associations at the file level.","marker":"[10]"},{"why":"Supplies the function parser used to validate extracted function names, docstrings, and code.","marker":"[11]"},{"why":"Defines the edit distance used to judge whether generated outputs align with the old or new reference.","marker":"[12]"},{"why":"Defines BM25 retrieval, used to select three similar training snippets for few-shot prompting.","marker":"[15]"},{"why":"Describes the dense 405-billion-parameter model family evaluated as one of the two baselines.","marker":"[16]"},{"why":"Describes the sparse mixture-of-experts 8x22B model family evaluated as the other baseline.","marker":"[17]"}],"fun_headline_variants":["LLMs struggle to keep code and docs in sync","New benchmark: LLMs align code-docs under 18% of time","CoDocBench: real commit pairs expose LLM maintenance gap","Code-docstring sync: LLMs left behind by real-world edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's value rests on the assumption that tracking same-named files on the main branch and detecting functions with regular expressions captures genuine coupled code-docstring changes without systematically missing larger ones.","fun_headline_variants_meta":{"raw":{"variants":["LLMs struggle to keep code and docs in sync","New benchmark: LLMs align code-docs under 18% of time","CoDocBench: real commit pairs expose LLM maintenance gap","Code-docstring sync: LLMs left behind by real-world edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000726,"raw_usage":{"total_tokens":3238,"prompt_tokens":914,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2250}},"tokens_in":530,"tokens_out":2324,"duration_ms":17332,"temperature":1.0,"reasoning_tokens":2250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:39:57.467284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A random-sample audit of commits where a function is moved to a differently named file, counting how often its docstring changes in the same commit, would settle whether the dataset's same-file tracking biases its samples; re-running the evaluations with a semantic similarity metric instead of edit distance would settle whether the reported difficulty is an artifact of the metric.","supporting_citations":[{"cited_title":"How documentation evolves over time,","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating estimate that docstrings are updated only about one-third as often as code."},{"cited_title":"Pydriller: Python framework for mining software repositories,","cited_arxiv_id":null,"evidence_quote":"Provides the Git-mining framework used to iterate over commits for each Python file."},{"cited_title":"Python tree-sitter,","cited_arxiv_id":null,"evidence_quote":"Provides the syntax-tree parser used to fix function-to-docstring associations at the file level."},{"cited_title":"The string-to-string correction problem,","cited_arxiv_id":null,"evidence_quote":"Defines the edit distance used to judge whether generated outputs align with the old or new reference."}],"review_version":1}