CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance

Kunal Pai; Premkumar Devanbu; Toufique Ahmed

REVIEW 4 major objections 4 minor 1 cited by

CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Real paired code-docstring edits expose an LLM alignment gap.

desk verdict A genuinely new dataset resource for code-docstring co-change, but the 'coupled' claim is never semantically validated and the evaluation lacks statistical grounding. read the letter →

arxiv 2502.00519 v2 pith:YK7ZWRE6 submitted 2025-02-01 cs.SE cs.LG

Kunal Pai , Premkumar Devanbu , Toufique Ahmed This is my paper · ORCID

classification cs.SEcs.LG

keywords code-documentationalignmentdocstringsoftwaremaintenancedatasetconstructionGitHubmininglargelanguagemodelscodegenerationbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CoDocBench, a dataset of 4,573 coupled changes mined from high-quality GitHub projects, where each sample is one commit in which a function's code and its docstring changed together. The authors' aim is to support two maintenance tasks: writing a docstring that matches new code, and writing code that matches a new docstring. On a 2,273-sample test split, they evaluate two large language models and report that correct temporal alignment of both old and new versions is rare, with the best model aligning both references in only 407 of 2,273 cases in the first task. This matters because documentation is known to lag behind code changes, so a natural benchmark of realistic paired edits could train and measure models that help developers keep documentation in sync.

What carries the argument

The carrying object is the CoDocBench dataset itself: each entry holds the old and new versions of a function's code and docstring, plus commit metadata, unified diffs, and file and function identifiers. The extraction machinery is a commit-diff pipeline that mines each file's commit history, detects function definitions and triple-quoted docstrings with regular expressions, and validates the function-name, code, and docstring associations with a syntax-tree parser and a function parser. The evaluation mechanism is edit distance: a generated output counts as correctly aligned when it is closer to the reference of its own temporal side than to the other side's reference. This machinery makes the benchmark reproducible and the difficulty claim measurable.

What would settle it

A random-sample audit of commits where a function is moved to a differently named file, counting how often its docstring changes in the same commit, would settle whether the dataset's same-file tracking biases its samples; re-running the evaluations with a semantic similarity metric instead of edit distance would settle whether the reported difficulty is an artifact of the metric.

Watch

Extended reading notes

Core claim

The central claim is that naturally occurring, coupled code-docstring changes can be extracted from real repositories at scale, and that this resource exposes a real capability gap in current large language models. The paper's construction pipeline takes Python files from popular, actively maintained projects, tracks each function across consecutive commits to the same file on the main branch, and keeps only entries where both the code and the docstring were edited together while excluding whitespace-only changes. Validation with a syntax-tree parser and a function parser confirms the function-docstring association. Under an edit-distance alignment criterion, the better model achieves 407 out of 2,273 correct 'both old and new' alignments for code/docstring generation, and at most 1,311 correct updates when the model is given the counterpart's new version, evidence that the tasks are feasible to pose but not yet solved.

Load-bearing premise

The dataset's value rests on the assumption that tracking same-named files on the main branch and detecting functions with regular expressions captures genuine coupled code-docstring changes without systematically missing larger ones.

Editorial extensions

If this is right

A reusable benchmark now exists for training and evaluating docstring-update and code-update models on realistic, commit-level paired edits.
The documented tendency for docstrings to lag code changes becomes a concrete target: models that pass CoDocBench-style alignment tests could help close that gap.
The asymmetric results, where docstring updates are easier than code updates and adding contextual information helps while BM25 few-shot retrieval mostly does not, give immediate design guidance for maintenance assistants.
The project-based 50/50 split makes the benchmark a tougher generalization test and supports future comparisons between models and methods.
The public release of the dataset and schema allows other researchers to add tasks such as commit-message generation or diff summarization using the same paired-change samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the dataset only follows functions inside same-named files on the main branch, its natural distribution is likely biased toward small, local edits; moves and renames that co-occur with docstring changes are systematically absent.
The edit-distance alignment criterion rewards surface-level similarity, so a semantic or execution-based metric could change the reported difficulty ranking.
The heavy project skew, with the top 25 projects supplying about 60% of samples, means reported accuracies may not transfer to less popular or differently documented codebases.
A direct test of the dataset's utility would be fine-tuning a smaller model on the training split and measuring whether alignment accuracy improves over the zero-shot baselines reported here.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

CoDocBench is a real, checkable new asset: 4,573 samples of Python functions whose code and docstring both changed in the same commit, mined from top GitHub projects via PyDriller and validated with Tree-Sitter. That target—same-commit code+docstring co-change—is not covered by existing summarization or search datasets, and the authors release code and data on Zenodo/GitHub, so the artifact itself is reproducible. The paper is honest about the main-branch and same-filename limitation. Credit where due: the construction pipeline is described in enough detail to re-implement, and the task framing (update code given new docstring, and vice versa) is a sensible way to operationalize documentation maintenance.

The soft spots are real, though. The biggest one: 'coupled' is never checked semantically. The pipeline only verifies that both the code string and the docstring string changed and parsed; it never checks that the docstring edit is about the same change as the code edit. A commit could fix a typo in a docstring and refactor code in the same commit, and that sample enters the dataset as a 'coupled change.' That contaminates the ground truth for both RQ1 and RQ2. The authors do not report any manual or automatic sampling to estimate what fraction of the 4,573 samples are genuinely aligned. This is a load-bearing assumption and it is unvalidated. The stress-test note is right: this affects every sample, not just the moved-function cases the authors acknowledge.

Second, the evaluation is under-powered: results are raw counts with no confidence intervals, no significance tests, and no chance baseline. For RQ1, the 'correct' definition requires both old and new alignments, but the paper doesn't report how often a trivial output (e.g., copying the input) would pass. Edit distance is a coarse proxy and is not validated against human judgment. Third, the dataset is small (4,573 samples, 60% from 25 projects), so the 'large' claim is modest.

Who is this for? Researchers working on docstring maintenance or LLM code editing who need a starting point and are willing to audit the data themselves. It deserves a serious referee because the resource is new and the authors are transparent, but the paper needs significant revision: add a semantic-coupling validation (manual sample, or an LLM-based filter), report statistical grounding, and calibrate chance. I'd send it to review, but with the expectation of heavy revision.

Referee Report

4 major / 4 minor

Summary. The paper introduces CoDocBench, a dataset of 4,573 Python code-docstring co-change samples mined from popular GitHub projects. The construction pipeline uses PyDriller over consecutive commits of same-named files on the main branch, detects function and docstring changes with regular expressions, filters to instances where both code and docstring changed (excluding whitespace-only edits), and validates parsing with Tree-Sitter and function_parser. The paper defines two tasks: RQ1 (temporal alignment of generated code/docstrings with old/new references) and RQ2 (generating an updated docstring from new code, or updated code from a new docstring). It evaluates Llama-3.1 405B Instruct Turbo and Mixtral 8x22B Instruct v0.1 under zero-shot, contextual-information, BM25 3-shot, and hybrid prompting, reporting raw counts of 'correct' alignments in Tables II and III. The authors conclude that current models find these maintenance-related tasks challenging. The paper also includes limitation statements about same-file tracking and main-branch-only mining.

Significance. CoDocBench addresses a real gap: there is no widely used, large benchmark specifically for coupled code-docstring evolution in software maintenance. If the dataset genuinely contains semantically coupled code and docstring changes, it provides a useful resource for training and evaluating models on documentation-update tasks, with a public release and concrete evaluation tasks. However, the significance is currently bounded by two unsupported assumptions: that syntactic co-change implies semantic coupling, and that raw edit-distance alignment counts are a valid measure of task success. If these are addressed, the dataset could be a valuable contribution to the software engineering and ML4Code communities.

major comments (4)

[Section II (Dataset Collection Methodology)] The pipeline establishes only syntactic co-change: both the code and the docstring changed in the same commit, and both parse correctly. It never validates that the docstring edit is semantically related to the code edit. The abstract and Section I define the dataset as containing 'coupled changes', but a commit with an unrelated docstring typo fix alongside a code refactor would pass the current filters. This matters for every sample in the 4,573-sample dataset and for the difficulty claims in Tables II and III, because the repository's new docstring is treated as the ground-truth aligned target for the new code. Please add a validation step, such as manual annotation of a random sample or a quantitative semantic-agreement check between the code diff and the docstring diff, and report the resulting precision. Without this, the central claim of coupling is unsupported.
[Section IV, Table II (RQ1 results)] For RQ1, a correct response requires both the old-generation-to-old-reference comparison and the new-generation-to-new-reference comparison to succeed. If each binary comparison is at chance (p=0.5), the expected accuracy for 'Aligned: Both' is 25%. The reported best count is 407/2273 = 17.9% for Llama-3.1 405B and 352/2273 = 15.5% for Mixtral, both below that chance level. This indicates that the models are not merely struggling but are likely systematically biased (for example, generating a single temporal style regardless of input), and the task should be described relative to this chance baseline rather than as an unqualified demonstration of difficulty. Please report chance baselines, per-condition percentages, and confidence intervals or significance tests for all counts in Tables II and III.
[Section IV (RQ2 evaluation definition)] The 'correct' criterion in RQ2 is based entirely on raw edit distance, with no validation that edit distance agrees with human judgments of whether the generated update matches the intended change. Edit distance is sensitive to surface form and tokenization, and the paper's own statistics in Figures 3-5 show that many code and docstring diffs are small, so the counts in Table III may partly reflect spurious surface similarity rather than genuine alignment with the new reference. Please validate the metric on a sample of outputs (for example, by human rating or a secondary semantic similarity measure) and report agreement; otherwise the interpretation of the raw counts is not reliable.
[Section II and Section V (Limitations)] The 'high-quality' characterization of the source projects is asserted rather than demonstrated. The selection criterion 'top 200 Python projects on GitHub' is not operationalized: no star threshold, activity cutoff, or candidate pool is given, and star count is a popularity signal rather than a direct quality measure. The same-file tracking limitation is acknowledged, and it is important because documentation updates that accompany function moves or renames are systematically excluded, which may bias the dataset toward small, local edits. Please quantify the extent of this exclusion if possible, and temper the 'natural' and 'high-quality' claims accordingly.

minor comments (4)

[Throughout] The name 'Schrek et al.' in the introduction is spelled 'Schreck et al.' in the reference list; please unify the spelling. Also, 'DocStrings' is capitalized inconsistently with 'docstring' elsewhere.
[Section III (Research Questions)] The exact prompt templates for RQ1, RQ2, and the prompting strategies are not given. Because the evaluation depends on the model outputs, the prompts should be included verbatim in an appendix for reproducibility.
[Section IV (Results)] Tables II and III report raw counts without the corresponding percentages or totals in the table cells. Adding percentages and including the 25% chance baseline for RQ1 would make the results much easier to interpret.
[Section IV (Models)] Generation settings such as temperature, top-p, max tokens, and number of decoding runs are not reported. Since edit-distance comparisons can be sensitive to sampling, these details should be stated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the dataset is curated from external GitHub history, and the model-difficulty results are evaluated against held-out repository references, not forced by the construction.

full rationale

CoDocBench is a resource paper, not a derivation chain, so the circularity burden is minimal. The curation pipeline (Section II) uses PyDriller, regular expressions, Tree-Sitter, and the CodeSearchNet function_parser to detect commits where both a function's code and its docstring changed; it filters to coupled-change samples and excludes whitespace-only edits. No parameter is fitted to the evaluation outcomes, and the benchmark tasks in Section III are defined by the authors but scored against repository-extracted references (old_ref and new_ref) using edit distance. Those references are external to the models and are not constructed from the models' outputs, so the reported difficulty in Tables II and III is not forced by the dataset definition. The alignment criteria in Section IV are evaluation metrics, not fitted constraints. The only author self-citation is [18], which is simply the dataset's release link and is not load-bearing. The skeptical concern that the pipeline never validates semantic coupling between code and docstring edits is a dataset-quality or label-noise issue, not a circularity step: the filter does not define the target answers as equivalent to its own outputs. No step reduces, by the paper's own equations or by self-citation, to its inputs, so the appropriate verdict is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or new physical or conceptual entities. Its load-bearing assumptions are about the validity of the edit-distance metric, the correctness of the extraction pipeline, and the representativeness of the project selection. These assumptions are reasonable for a first resource paper but are not fully validated.

assumptions (3)

domain assumption Edit distance to the old or new reference is a valid measure of correct alignment and update quality.
Used to define success in RQ1 and RQ2 (Section IV); the paper notes that future work could use other metrics, but it does not validate that lower edit distance corresponds to developer-useful updates.
domain assumption The regex and Tree-sitter pipeline correctly identifies functions and coupled changes from consecutive commits on the main branch.
Dataset construction relies on detecting 'def' lines, triple-quoted docstrings, and same file names across commits (Section II); the Limitations section admits moves and renames break tracking.
domain assumption Top-200 high-star, actively maintained Python projects provide representative, high-quality samples.
Project selection criteria are star count and recent commit activity (Section II); this is a proxy for code quality, not direct evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance." pith.science (2026). https://pith.science/paper/YK7ZWRE6

@misc{pith2026250200519,
  author       = {Pith},
  title        = {Pith review of: CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YK7ZWRE6}},
  note         = {Machine review of arXiv:2502.00519}
}

abstract

One of the central tasks in software maintenance is being able to understand and develop code changes. Thus, given a natural language description of the desired new operation of a function, an agent (human or AI) might be asked to generate the set of edits to that function to implement the desired new operation; likewise, given a set of edits to a function, an agent might be asked to generate a changed description, of that function's new workings. Thus, there is an incentive to train a neural model for change-related tasks. Motivated by this, we offer a new, "natural", large dataset of coupled changes to code and documentation mined from actual high-quality GitHub projects, where each sample represents a single commit where the code and the associated docstring were changed together. We present the methodology for gathering the dataset, and some sample, challenging (but realistic) tasks where our dataset provides opportunities for both learning and evaluation. We find that current models (specifically Llama-3.1 405B, Mixtral 8$\times$22B) do find these maintenance-related tasks challenging.

Figures

Figures reproduced from arXiv: 2502.00519 by the authors.

**Figure 2.** Workflow for creating CoDocBench. The figure illustrates a multi-stage process: (A) Collect all Python files per project, (B) Commit processing per Python file, (C) Using regular expressions for pairwise commit change detection per function, (D) Using AST and post-processing to refine detected changes and create CoDocBench, and (E) Generating a test and training set through a random 50/50 project split. having a hig… view at source ↗

**Figure 5.** Scatterplot showing length of code diff (in tokens) vs. [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Cumulative percentage distribution of dataset entries [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

HASHIRU: Hierarchical Agent System for Hybrid Intelligent Resource Utilization
cs.MA 2025-06 conditional novelty 4.0 of 10

A hierarchical AI agent framework that dynamically hires and fires specialist models and creates tools reports strong benchmark numbers, though its gains may come from tool use rather than the architecture.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages · cited by 1 Pith paper

[1]

Software maintenance,

G. Canfora and A. Cimitile, “Software maintenance,” in Handbook of Software Engineering and Knowledge Engineering: V olume I: Funda- mentals. World Scientific, 2001, pp. 91–120

work page 2001
[2]

Automated program repair in the era of large pre-trained language models,

C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1482–1494

work page 2023
[3]

Resolving code review comments with machine learning,

A. Frömmgen, J. Austin, P. Choy, N. Ghelani, L. Kharatyan, G. Surita, E. Khrapko, P. Lamblin, P.-A. Manzagol, M. Revaj et al. , “Resolving code review comments with machine learning,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice , 2024, pp. 204–215

work page 2024
[4]

Large sequence models for software development activities,

P. Maniatis and D. Tarlow, “Large sequence models for software development activities,” https://research.google/blog/ large-sequence-models-for-software-development-activities/

work page
[5]

Autocoderover: Autonomous program improvement,

Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592–1604

work page 2024
[6]

An overview of industrial software documentation practice,

M. Visconti and C. R. Cook, “An overview of industrial software documentation practice,” in 12th International Conference of the Chilean Computer Science Society, 2002. Proceedings. IEEE, 2002, pp. 179– 186

work page 2002
[7]

How documentation evolves over time,

D. Schreck, V . Dallmeier, and T. Zimmermann, “How documentation evolves over time,” in Ninth international workshop on Principles of software evolution: in conjunction with the 6th ESEC/FSE joint meeting , 2007, pp. 4–10

work page 2007
[8]

Pydriller: Python framework for mining software repositories,

D. Spadini, M. Aniche, and A. Bacchelli, “Pydriller: Python framework for mining software repositories,” in Proceedings of the 2018 26th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering , 2018, pp. 908– 911

work page 2018

Show all 18 references

[9]

O’Reilly Media, Inc

J. Friedl, Mastering regular expressions. " O’Reilly Media, Inc.", 2006

2006
[10]

Python tree-sitter,

Tree-sitter, “Python tree-sitter,” https://github.com/tree-sitter/ py-tree-sitter, 2024

2024
[11]

Codesearchnet challenge: Evaluating the state of semantic code search,

H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

1909 arXiv
[12]

The string-to-string correction problem,

R. A. Wagner and M. J. Fischer, “The string-to-string correction problem,” J. ACM , vol. 21, no. 1, p. 168–173, Jan. 1974. [Online]. Available: https://doi.org/10.1145/321796.321811

1974
[13]

A guided tour to approximate string matching,

G. Navarro, “A guided tour to approximate string matching,” ACM computing surveys (CSUR) , vol. 33, no. 1, pp. 31–88, 2001

2001
[14]

Binary codes capable of correcting deletions, insertions, and reversals,

V . Levenshtein, “Binary codes capable of correcting deletions, insertions, and reversals,” Proceedings of the Soviet physics doklady , 1966

1966
[15]

The probabilistic relevance frame- work: Bm25 and beyond,

S. Robertson, H. Zaragoza et al. , “The probabilistic relevance frame- work: Bm25 and beyond,” F oundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009

2009
[16]

The llama 3 herd of models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

2024 arXiv
[17]

Mixtral of experts,

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

2024 arXiv
[18]

CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance,

K. Pai, P. Devanbu, and T. Ahmed, “CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance,” Nov. 2024. [Online]. Available: https://github.com/kunpai/codocbench

2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Software maintenance,

G. Canfora and A. Cimitile, “Software maintenance,” in Handbook of Software Engineering and Knowledge Engineering: V olume I: Funda- mentals. World Scientific, 2001, pp. 91–120

work page 2001

[2] [2]

Automated program repair in the era of large pre-trained language models,

C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1482–1494

work page 2023

[3] [3]

Resolving code review comments with machine learning,

A. Frömmgen, J. Austin, P. Choy, N. Ghelani, L. Kharatyan, G. Surita, E. Khrapko, P. Lamblin, P.-A. Manzagol, M. Revaj et al. , “Resolving code review comments with machine learning,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice , 2024, pp. 204–215

work page 2024

[4] [4]

Large sequence models for software development activities,

P. Maniatis and D. Tarlow, “Large sequence models for software development activities,” https://research.google/blog/ large-sequence-models-for-software-development-activities/

work page

[5] [5]

Autocoderover: Autonomous program improvement,

Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592–1604

work page 2024

[6] [6]

An overview of industrial software documentation practice,

M. Visconti and C. R. Cook, “An overview of industrial software documentation practice,” in 12th International Conference of the Chilean Computer Science Society, 2002. Proceedings. IEEE, 2002, pp. 179– 186

work page 2002

[7] [7]

How documentation evolves over time,

D. Schreck, V . Dallmeier, and T. Zimmermann, “How documentation evolves over time,” in Ninth international workshop on Principles of software evolution: in conjunction with the 6th ESEC/FSE joint meeting , 2007, pp. 4–10

work page 2007

[8] [8]

Pydriller: Python framework for mining software repositories,

D. Spadini, M. Aniche, and A. Bacchelli, “Pydriller: Python framework for mining software repositories,” in Proceedings of the 2018 26th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering , 2018, pp. 908– 911

work page 2018