Pith. sign in

REVIEW 4 major objections 5 minor 86 references

Evaluating the Use of LLMs for Documentation to Code Traceability

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large language models can map documentation to code with F1 scores near 80%, roughly doubling the best classical retrieval baselines on the two studied projects.

desk verdict A careful empirical study that overclaims its absolute F1 because the ground-truth artifact lists are incomplete and zero-link segments are dropped, but the relative LLM-vs-baseline result and the new datasets are worth taking seriously. read the letter →

arxiv 2506.16440 v1 pith:C2XV2VPJ submitted 2025-06-19 cs.SE cs.AIcs.LG

Ebube Alor , SayedHassan Khatoonabadi , Emad Shihab This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords softwaretraceabilitydocumentation-to-codelargelanguagemodelstracelinkrecoveryone-to-manymatchingLLMevaluationinformationretrievalbaselines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models, given a curated list of code artifacts, can identify which source-code elements a documentation segment refers to, and can do so accurately enough to serve as a first-pass traceability tool. On two manually built datasets from post-2024 open-source projects, the best model reaches F1 of 79.4% and 80.4%, while the best baseline (TF-IDF or CodeBERT) reaches 54.2% and 69.3%. The study also claims the limits: relationship explanations are substantively right in over 97% of cases but fully correct only 42.9% to 71.1% of the time, and multi-step trace chains rarely lose both endpoints but often miss intermediate elements. The author argues that task framing, specifically a one-to-many matching strategy, is critical to the observed performance.

What carries the argument

The central mechanism is the one-to-many matching strategy: each documentation segment is sent to the LLM once along with the complete, manually curated list of candidate code artifacts and the repository directory tree, and the model returns the subset of artifacts that relate to the segment, with relationship descriptions and trace chains. This design avoids the output-token bottleneck of many-to-many prompts, which the paper shows drops recall below 3% and F1 to about 5%. The other load-bearing component is the five-step dataset pipeline: heading-based segmentation, artifact extraction at class/method/statement granularity, manual trace-link creation, a second verification pass with Cohen's kappa of 0.94, and selection of projects created after the LLMs' knowledge cutoffs to reduce contamination risk.

What would settle it

Re-run the same prompts on the same datasets but replace the curated artifact list with the full set of all files in each repository; if precision or recall drops substantially, the measured F1 is an artifact of the closed candidate set. Also, include documentation segments with no valid links (which the dataset pipeline excluded) and measure false positives.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art LLMs substantially outperform traditional information-retrieval baselines in identifying documentation-to-code trace links on unseen, small-to-medium codebases. With a one-to-many prompt holding the documentation segment and the full artifact list, Claude 3.5 Sonnet achieves F1 of 79.4% on Crawl4AI and 80.4% on Unity Catalog, compared with best baseline F1 of 54.2% and 69.3%. Precision is consistently high (above 87%), while recall varies by model from 47% to 75%. The explicit- and implicit-link finding is stronger than the explanation finding: relaxed explanation accuracy, counting partially correct explanations, exceeds 97% for every model and dataset, but strict explanation accuracy ranges from 42.9% to 71.1%. For trace chains, complete matches range from about 13% to 80% depending on dataset, with endpoint loss below 2% across all pairs.

Load-bearing premise

The manually constructed ground-truth links are complete and correct, and the candidate artifact list contains every code element the documentation can legitimately refer to; if either fails, the reported F1 and relaxed accuracy will not transfer to practice.

Editorial extensions

If this is right

  • Development teams can adopt LLMs as first-pass trace discovery with F1 near 80% on documentation styles similar to these projects, far above the 36% to 69% of baselines.
  • Tools should use one-to-many prompting: many-to-many dropped recall below 3% in the paper's experiments.
  • Human review remains necessary for explanations and trace-chain interiors, since strict explanation accuracy is below 72% and complete chains are below 80%.
  • Chain endpoints are reliable anchors, with fewer than 2% of chains losing both endpoints.
  • Documentation style dictates which cues drive recall: explicit mentions in narrative docs and parameter/return tables in API docs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported F1 depends on giving the LLM the exact curated artifact list; on a large codebase, the candidate set would have to be pruned, and end-to-end F1 could be lower than 80%.
  • Phantom-link false positives could be filtered automatically because the paper shows the LLM sometimes references artifacts absent from the provided list; a simple check against the list would remove those errors.
  • The one-to-many advantage is likely tied to output-token limits; models with larger generation budgets may allow many-to-many prompts without the same collapse, so the framing recommendation may age.
  • A cost-aware extension would be to batch several segments per prompt while keeping per-segment output sections; the paper did not test this middle ground.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical evaluation of three LLMs (Claude 3.5 Sonnet, GPT-4o, o3-mini) for documentation-to-code traceability, introducing two new datasets from post-cutoff projects (Crawl4AI, Unity Catalog). The study reports three sets of findings: RQ1 reports that the best LLM achieves F1-scores of 79.4% and 80.4%, substantially outperforming TF-IDF, BM25, and CodeBERT baselines; RQ2 reports relationship-explanation quality with strict accuracy 42.9–71.1% and relaxed accuracy exceeding 97%; RQ3 reports multi-step trace-chain recovery with high endpoint accuracy but variable intermediate-element identification. The authors also analyze error patterns and study the effect of task framing (matching strategy and added file context), concluding that LLMs are promising but require human-in-the-loop tool design.

Significance. If the numerical results were valid, this would be a useful contribution: the datasets are post-training-cutoff, the RQ1 protocol is rigorous (five shuffled runs, inter-rater agreement of 0.94, Scott-Knott ESD, optimistic threshold tuning for baselines), and the replication package is public. The error taxonomy and the one-to-many vs. many-to-many task-framing analysis provide practical guidance for tool builders. However, the headline F1 values and the abstract's claim of substantial superiority over baselines are undermined by ground-truth completeness issues in Section 3.3 that carry into the reported metrics, as detailed in the major comments. The core relative ranking of LLM vs. baseline may survive correction, but the absolute numbers as stated are not established.

major comments (4)
  1. [§3.3, Step 1 and Table 1] The dataset creation excludes all documentation segments marked as having 'zero links' (e.g., the 'Newsletter Signup' segment in the running example). This means the RQ1 precision/recall/F1 values in Table 2 are computed only over segments that are known to contain at least one trace link; the evaluation never measures the models' ability to reject irrelevant documentation. A practical traceability tool must also decide 'no link exists,' and that decision is a nontrivial part of precision. The headline F1 values therefore overstate the operational accuracy of the LLMs. Please either include zero-link segments with their 'no-link' ground truth, or explicitly qualify all reported F1/precision/recall as conditional on positive segments and adjust the abstract accordingly.
  2. [§3.3, Step 2 and §4.3, Table 3] The Phantom Link (PAL) error category is defined as tracing to code artifacts 'absent from the provided artifacts,' and the example NlpSentenceChunking.chunk() is explicitly named in the documentation. The paper does not establish that this symbol is absent from the repository—only that it was not in the curated artifact list. If such methods exist in the codebase, the ground-truth label 'false positive' is an artifact of incomplete dataset construction, and the ground-truth trace links are incomplete. Since all F1, precision, and recall values in Table 2 are computed against this ground truth, the absolute numbers 79.4% and 80.4% are not valid estimates of traceability accuracy. The relative ordering may be defensible because baselines receive the same candidate set, but the specific numeric claims in the abstract and RQ1 require either completing the artifact list after a documented scan of documentation-mentioned symbols or re-framing the results as 'traceability on the provided candidate artifact set.'
  3. [§5.2 and Appendix, 'Prompt for LLM as a Judge'] The RQ2 relaxed-accuracy claim (exceeding 97%) relies on an LLM-as-a-judge prompt that explicitly instructs the judge to 'Do NOT penalize minor omissions about code details or parameters' and to classify as partially_correct only if the predicted text 'contradicts or significantly misunderstands a major aspect' of the ground truth. This leniency bias, combined with the use of Claude Sonnet 3.5 as the judge for explanations generated by Claude Sonnet 3.5 among others, likely inflates both strict and relaxed accuracy. The paper reports that a representative subset was manually reviewed, but it does not give the judge-human agreement. Please report inter-rater agreement between the LLM judge and human annotators, and re-analyze RQ2 with a judge prompt that does not encode a strong prior toward leniency.
  4. [§6.3, Table 7 and Appendix Listing 1] The RQ3 claim that 'endpoints are reliable anchors' is to a large extent a consequence of the prompt design rather than an empirical discovery. The prompt (Listing 1, instructions) explicitly requires chains to 'Always start with the document name' and 'Must end at the traced artifact,' and the chain analysis in §6.2 starts from trace links already identified as true positives in RQ1. Thus the starting endpoint is fixed to the document name and the ending endpoint is correct by construction of the TP set; the >98% endpoint accuracy does not measure the LLM's independent capability to recover chain endpoints. Please either test chain construction without prescribing the endpoints, or remove/weaken the 'reliable anchors' claim and present the endpoint statistics only as a sanity check.
minor comments (5)
  1. [Abstract and §4.3] The abstract states that the best LLM 'substantially outperforms' baselines whose best F1-scores were 54.2% and 69.3%; however, in Table 2 GPT-4o's Unity Catalog F1 (69.3%) exactly ties the TF-IDF baseline, so the 'substantially outperforming' phrasing is only strictly true for Claude Sonnet 3.5.
  2. [§3.1] The selection criteria say all chosen LLMs must have 'Recent and Verifiable Knowledge Cutoff,' yet o3-mini is described as 'also later included'; please provide a consistent justification for its inclusion.
  3. [Table 2 and §4.3] The sentence 'Method-level tracing generally achieves higher F1 scores across the models and datasets compared to class-level tracing' is contradicted by the Unity Catalog rows where Claude Sonnet 3.5 achieves 98.6% class-level vs. 78.6% method-level; the later 'exceptions exist' caveat should be moved before the generalization.
  4. [§5.2] The grammar of 'we analyzed whether the LLMs' could detect intermediate elements' (also in §6.2) should be corrected to 'whether the LLMs could detect.'
  5. [§9.4] The paper states that all runs were performed within a defined time window but does not report the dates or the API model version snapshots used for o3-mini; please add this reproducibility detail.

Circularity Check

2 steps flagged · score 6.0 of 10

RQ3 endpoint anchoring and RQ2 relaxed accuracy reduce to prompt/rubric construction; the central RQ1 F1 comparison is non-circular.

  1. self definitional [Section 6.3 (Trace Chain Analysis) and Appendix Listing 1, prompt instruction 12]
    "For traceability pathways: - Always start with the document name (extracted from location) ... - Must end at the traced artifact ... This is expected as we provide the LLM with the starting and end points in the prompt."

    The prompt explicitly instructs the LLM that every trace chain must 'Always start with the document name' and 'Must end at the traced artifact.' RQ3 then measures whether chains preserve their endpoints and reports 'Endpoints are reliable anchors' with 'below 2% incorrect.' The paper itself concedes, 'This is expected as we provide the LLM with the starting and end points in the prompt.' Thus the endpoint-accuracy result is an artifact of the output-format instruction, not an empirical finding about the LLM's ability to reconstruct multi-step dependency chains. The prefix/suffix partial-match categories are likewise constructed from endpoints the model was forced to emit.

  2. self definitional [Section 5.2 (Relationship Explanation) and Appendix Listing 2 (LLM-as-judge prompt)]
    "relaxed accuracy, the percentage classified as either correct or partially correct ... Do NOT penalize minor omissions about code details or parameters. Only label as partially_correct if the predicted text contradicts or significantly misunderstands a major aspect of the ground truth. Everything else can be correct ... Treat missing details or code specifics as unimportant, as long as the main relationship is correct."

    The relaxed-accuracy metric is defined as the share of explanations judged 'correct' or 'partially correct,' and the judge prompt defines 'correct' extremely broadly: 'Do NOT penalize minor omissions,' 'Everything else can be correct,' and 'Treat missing details or code specifics as unimportant.' With this rubric, any explanation that preserves the high-level relationship qualifies, so the reported finding that 'relaxed accuracy exceeds 97% across all LLMs' is substantially entailed by the evaluation instructions rather than by independent measurement of explanation quality. The judge is also Claude 3.5 Sonnet, the same model family as the best-performing evaluated LLM, which compounds the concern.

full rationale

The central RQ1 result—LLMs achieving F1 79.4%/80.4% and outperforming TF-IDF, BM25, and CodeBERT—rests on a manually verified ground truth (Cohen's kappa = 0.94) and newly created post-cutoff datasets, so it does not reduce to its own inputs. The baseline threshold grid search is disclosed and is not a circular step. However, two of the paper's three headline capability claims are substantially constructed: RQ3's 'endpoints are reliable anchors' follows from the prompt's instruction that chains always start with the document name and end at the traced artifact (as the paper itself states), and RQ2's relaxed accuracy >97% follows from a judge rubric that instructs the evaluator to classify almost everything as correct or partially correct. These are self-definitional findings rather than independent measurements. The 'phantom link' error category also defines false positives relative to the curated artifact list, so some labeled errors may be labeling omissions; this is a ground-truth completeness threat to the absolute F1 values rather than a circular reduction, and it does not favor the LLMs' headline claim. Overall, the main comparison is non-circular, but the paper's secondary capability claims exhibit partial circularity by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical evaluation, so its load-bearing inputs are dataset-construction decisions and evaluation assumptions rather than mathematical axioms. The central claim does not depend on fitted free parameters for the LLMs; the only fitted parameters are the baseline thresholds, which are tuned to give the baselines an advantage.

free parameters (2)
  • Baseline similarity thresholds (TF-IDF, BM25, CodeBERT) = Per-method, per-dataset thresholds from grid search in 0.05 increments maximizing F1
    Section 3.4, Baseline Methods: thresholds selected post hoc on the evaluation set. Disclosed by the authors as an optimistic estimate of baseline performance.
  • o3-mini reasoning_effort = high
    Section 3.1, LLM Settings: explicitly set to high for o3-mini, a hand-chosen hyperparameter that affects that model's results.
assumptions (4)
  • domain assumption Documentation segments with zero valid trace links were removed from the datasets.
    Section 3.3 Step 1: segments that lacked any links to code were 'marked as having zero links and excluded from the dataset'. This assumes the deployment setting also lacks no-link segments, which is unlikely in practice.
  • domain assumption Manual trace-link labeling by the two authors (Cohen's kappa = 0.94) is the correct ground truth for all 800 links.
    Section 3.3 Steps 3 and 4: if the ground truth misses implicit links, precision, recall, and the error taxonomy are all affected.
  • domain assumption Projects created after May 1, 2024 (Unity Catalog, Crawl4AI) are absent from the LLMs' pretraining data.
    Sections 3.1 to 3.2: the no-contamination guarantee rests on repository creation dates, a proxy the authors acknowledge as imperfect in Section 9.1.
  • domain assumption LLM-as-a-judge (Claude 3.5 Sonnet) labels relationship explanations as correct, partially correct, or incorrect accurately.
    Section 5.2: only a 'representative subset' was manually reviewed, and the manual agreement rate is not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating the Use of LLMs for Documentation to Code Traceability." pith.science (2026). https://pith.science/paper/C2XV2VPJ

@misc{pith2026250616440,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Use of LLMs for Documentation to Code Traceability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C2XV2VPJ}},
  note         = {Machine review of arXiv:2506.16440}
}
read the original abstract

Large Language Models (LLMs) offer new potential for automating documentation-to-code traceability, yet their capabilities remain underexplored. We present a comprehensive evaluation of LLMs (Claude 3.5 Sonnet, GPT-4o, and o3-mini) in establishing trace links between various software documentation (including API references and user guides) and source code. We create two novel datasets from two open-source projects (Unity Catalog and Crawl4AI). Through systematic experiments, we assess three key capabilities: (1) trace link identification accuracy, (2) relationship explanation quality, and (3) multi-step chain reconstruction. Results show that the best-performing LLM achieves F1-scores of 79.4% and 80.4% across the two datasets, substantially outperforming our baselines (TF-IDF, BM25, and CodeBERT). While fully correct relationship explanations range from 42.9% to 71.1%, partial accuracy exceeds 97%, indicating that fundamental connections are rarely missed. For multi-step chains, LLMs maintain high endpoint accuracy but vary in capturing precise intermediate links. Error analysis reveals that many false positives stem from naming-based assumptions, phantom links, or overgeneralization of architectural patterns. We demonstrate that task-framing, such as a one-to-many matching strategy, is critical for performance. These findings position LLMs as powerful assistants for trace discovery, but their limitations could necessitate human-in-the-loop tool design and highlight specific error patterns for future research.

Figures

Figures reproduced from arXiv: 2506.16440 by the authors.

Figure 1
Figure 1. Five-step dataset-creation pipeline. [57, 58]. It features a large 200k token context window. It also has an October 2023 knowledge cutoff like GPT-4o. 3.2 Project Selection To evaluate the performance of LLMs in documentation-to-code traceability tasks, we required datasets containing software projects with well-documented codebases that were unlikely to be part of the selected LLMs’ training data. Given that the l… view at source ↗
Figure 2
Figure 2. Illustrative documentation with two segments. The first segment (above the line) discusses concurrency [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Illustrative Python code for the Crawler class, which is referenced by Segment A. example, we preserved it for further analysis. Meanwhile, the “Newsletter Signup” text (Segment B) was discarded as it did not discuss any relevant information. Step 2: Code Artifact Organization. Concurrently, we identified and cataloged relevant code elements, termed code artifacts. We established the granularity (level of detail) fo… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustrative structure of a dataset entry using the running example. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Simplified prompt structure for documentation-to-code traceability. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

86 extracted references · 61 canonical work pages

  1. [1]

    Bassem Abid, Jane Cleland-Huang, and Ying Zou. 2014. Using traceability links to recommend adaptive changes for documentation evolution.IEEE Transactions on Software Engineering40, 10 (2014), 997–1013

    2014

  2. [2]

    Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shep- herd. 2020. Software documentation: the practitioners’ perspective. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 590–601

    2020

  3. [3]

    Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

    2019

  4. [4]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2024. Can LLMs replace manual annotation of software engineering artifacts?arXiv preprint arXiv:2408.05534(2024)

    arXiv 2024

  5. [5]

    Syed Juned Ali, Varun Naganathan, and Dominik Bork. 2024. Establishing Traceability Between Natural Language Requirements and Software Artifacts by Combining RAG and LLMs. InInternational Conference on Conceptual Modeling. Springer, 295–314

    2024

  6. [6]

    Anthropic. [n. d.]. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview

  7. [7]

    AI Anthropic. 2024. Claude 3.5 sonnet model card addendum.Claude-3.5 Model Card3, 6 (2024)

    2024

  8. [8]

    Giuliano Antoniol, Gerardo Canfora, Gerardo Casazza, Andrea De Lucia, and Ettore Merlo. 2002. Recovering traceability links between code and documentation.IEEE transactions on software engineering28, 10 (2002), 970–983

    work page 2002

Show all 86 references
  1. [9]

    Daniel Blasco, Carlos Cetina, and Óscar Pastor. 2020. A fine-grained requirement traceability evolutionary algorithm: Kromaia, a commercial video game case study.Information and Software Technology119 (2020), 106235

    2020

  2. [10]

    Héctor Cadavid, Vasilios Andrikopoulos, and Paris Avgeriou. 2022. Documentation-as-code for interface control document management in systems of systems: A technical action research study. InEuropean Conference on Software Architecture. Springer, 19–37

    2022

  3. [11]

    Sandra Casas, Diana Cruz, Graciela Vidal, and Marcela Constanzo. 2021. Uses and applications of the OpenAPI/Swagger specification: a systematic mapping of the literature. In2021 40th International Conference of the Chilean Computer Science Society (SCCC). IEEE, 1–8

    2021

  4. [12]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    2021 arXiv

  5. [13]

    Xiaofan Chen and John Grundy. 2011. Improving automated documentation to code traceability by combining retrieval techniques. In2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). IEEE, 223–232

    2011

  6. [14]

    Jane Cleland-Huang. 2011. Traceability in agile projects. InSoftware and Systems Traceability. Springer, 265–275

    2011

  7. [15]

    Jane Cleland-Huang, Adam Czauderna, Marek Gibiec, and John Emenecker. 2010. A machine learning approach for tracing regulatory codes to product specific requirements. InProceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. 155–164

    2010

  8. [16]

    2012.Software and systems traceability

    Jane Cleland-Huang, Orlena Gotel, Andrea Zisman, et al. 2012.Software and systems traceability. Vol. 2. Springer

    2012

  9. [17]

    Jane Cleland-Huang, Orlena CZ Gotel, Jane Huffman Hayes, Patrick Mäder, and Andrea Zisman. 2014. Software traceability: trends and future directions. InFuture of software engineering proceedings. 55–69

    2014

  10. [18]

    Andrea De Lucia, Fausto Fasano, Rocco Oliveto, and Genoveffa Tortora. 2006. Can information retrieval techniques effectively support traceability link recovery?. In14Th IEEE international conference on program comprehension (ICPC’06). IEEE, 307–316

    2006

  11. [19]

    Google Deepmind. [n. d.]. Google Gemini 2.0 Pro. https://deepmind.google/technologies/gemini/pro/

  12. [20]

    Wei Ding, Peng Liang, Antony Tang, and Hans Van Vliet. 2014. Knowledge-based approaches in software documentation: A systematic literature review.Information and Software Technology56, 6 (2014), 545–567

    2014

  13. [21]

    Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. A comparative analysis of large language models for code documentation generation. InProceedings of the 1st ACM International Conference on AI-Powered Software. 65–73

    2024

  14. [22]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on , Vol. 1, No. 1, Article . Publication date: Septem...

    2023

  15. [23]

    Leye Feng, Lin Ma, and Xin Zhang. 2024. Recovering Traceability Links between Release Notes and Related Software Artifacts. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. IEEE, 1–4

    2024

  16. [24]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al . 2020. Codebert: A pre-trained model for programming and natural languages.arXiv preprint arXiv:2002.08155(2020)

    2020 arXiv

  17. [25]

    Andrew Forward and Timothy C Lethbridge. 2002. The relevance of software documentation, tools and technologies: a survey. InProceedings of the 2002 ACM symposium on Document engineering. 26–33

    2002

  18. [26]

    Dominik Fuchß, Haoyu Liu, Tobias Hey, Jan Keim, and Anne Koziolek. 2025. Enabling architecture traceability by llm-based architecture component name extraction. In2025 IEEE 22nd International Conference on Software Architecture (ICSA). IEEE, 1–12

    2025

  19. [27]

    Orlena Gotel, Jane Cleland-Huang, Jane Huffman Hayes, Andrea Zisman, Alexander Egyed, Paul Grünbacher, Alex Dekhtyar, Giuliano Antoniol, and Jonathan Maletic. 2012. The grand challenge of traceability (v1. 0).Software and systems traceability(2012), 343–409

    2012

  20. [28]

    Jin Guo, Jinghui Cheng, and Jane Cleland-Huang. 2017. Semantically enhanced software traceability using deep learning techniques. In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 3–14

    2017

  21. [29]

    Jameleddine Hassine. 2024. An llm-based approach to recover traceability links between security requirements and goal models. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 643–651

    2024

  22. [30]

    Jane Huffman Hayes, Alex Dekhtyar, and James Osborne. 2003. Improving requirements tracing via information retrieval. InProceedings. 11th IEEE International Requirements Engineering Conference, 2003.IEEE, 138–147

    2003

  23. [31]

    Jane Huffman Hayes, Alex Dekhtyar, and Senthil Karthikeyan Sundaram. 2006. Advancing candidate link generation for requirements tracing: The study of methods.IEEE Transactions on Software Engineering32, 1 (2006), 4–19

    2006

  24. [32]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  25. [33]

    Hao Hu, Haiyang Wang, and Qing Wang. 2017. Refining Traceability Links between Code and Software Documents. InProceedings of the 9th Asia-Pacific Symposium on Internetware. ACM, 1–9

    2017

  26. [34]

    Hao Hu, Haiyang Wang, and Qing Wang. 2018. Toward accurate link between code and software documentation. Science China Information Sciences61, 1 (2018), 1–3

    2018

  27. [35]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    2024 arXiv

  28. [36]

    Nikitas Karanikolas, Eirini Manga, Nikoletta Samaridi, Eleni Tousidou, and Michael Vassilakopoulos. 2023. Large language models versus natural language understanding and generation. InProceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics. 278–290

    2023

  29. [37]

    Junaed Younus Khan and Gias Uddin. 2022. Automatic code documentation generation using gpt-3. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–6

    2022

  30. [38]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.biometrics (1977), 159–174

    1977

  31. [39]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics12 (2024), 157–173

    2024

  32. [40]

    Andrea De Lucia, Fausto Fasano, Rocco Oliveto, and Genoveffa Tortora. 2007. Recovering traceability links in software artifact management systems using information retrieval methods.ACM Transactions on Software Engineering and Methodology (TOSEM)16, 4 (2007), 13–es

    2007

  33. [41]

    Yuzhan Ma, Sarah Fakhoury, Michael Christensen, Venera Arnaoudova, Waleed Zogaan, and Mehdi Mirakhorli. 2018. Automatic classification of software artifacts in open-source applications. InProceedings of the 15th International Conference on Mining Software Repositories. 414–425

    2018

  34. [42]

    Walid Maalej and Martin P Robillard. 2013. Patterns of knowledge in API reference documentation.IEEE Transactions on software Engineering39, 9 (2013), 1264–1282

    2013

  35. [43]

    Ana C Marcén, Raúl Lapeña, Oscar Pastor, and Carlos Cetina. 2020. Traceability link recovery between requirements and models using an evolutionary algorithm guided by a learning to rank algorithm: Train control and management case.Journal of Systems and Software163 (2020), 110519

    2020

  36. [44]

    Andrian Marcus and Jonathan I Maletic. 2003. Recovering documentation-to-source-code traceability links using latent semantic indexing. In25th International Conference on Software Engineering, 2003. Proceedings.IEEE, 125–135. , Vol. 1, No. 1, Article . Publication date: Septem...

    2003

  37. [45]

    Arsalan Masoudifard, Mohammad Mowlavi Sorond, Moein Madadi, Mohammad Sabokrou, and Elahe Habibi. 2024. Leveraging Graph-RAG and Prompt Engineering to Enhance LLM-Based Automated Requirement Traceability and Compliance Checks.arXiv preprint arXiv:2412.08593(2024)

    2024 arXiv

  38. [46]

    Masri, Maroun M

    Wail S. Masri, Maroun M. Diab, and Elia Abu Shama. 2018. Mining and discovery of hidden relationships between software source codes and related textual documents.Digital Scholarship in the Humanities33, 2 (2018), 261–279

    2018

  39. [47]

    Chris Mills. 2017. Automating traceability link recovery through classification. InProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 1068–1070

    2017

  40. [48]

    Chris Mills, Javier Escobar-Avila, and Sonia Haiduc. 2018. Automatic traceability maintenance via machine learning classification. In2018 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 369–380

    2018

  41. [49]

    Laura Moreno, Andrian Marcus, Lori Pollock, and K Vijay-Shanker. 2013. Jsummarizer: An automatic generator of natural language summaries for java classes. In2013 21st International Conference on Program Comprehension (ICPC). IEEE, 230–232

    2013

  42. [50]

    Nachiappan Nagappan, Brendan Murphy, and Victor Basili. 2008. The influence of organizational structure on software quality: an empirical case study. InProceedings of the 30th international conference on Software engineering. 521–530

    2008

  43. [51]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  44. [52]

    Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. 2024. Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825 (2024)

    2024 arXiv

  45. [53]

    Marc North, Amir Atapour-Abarghouei, and Nelly Bencomo. 2024. Code gradients: Towards automated traceability of llm-generated code. In2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 321–329

    2024

  46. [54]

    OpenAI. [n. d.]. https://platform.openai.com

  47. [55]

    OpenAI. 2025. Introducing GPT-4.1 in the API | openai. https://openai.com/index/gpt-4-1/

    2025

  48. [56]

    OpenAI. 2025. Models. https://platform.openai.com/docs/models/gpt-4o [Accessed 04-03-2025]

    2025

  49. [57]

    OpenAI. 2025. Models. https://platform.openai.com/docs/models/o3-mini [Accessed 04-03-2025]

    2025

  50. [58]

    OpenAI. 2025. OpenAI o3-mini. https://openai.com/index/openai-o3-mini [Accessed 04-03-2025]

    2025

  51. [59]

    Ipek Ozkaya. 2023. Application of large language models to software engineering tasks: Opportunities, risks, and implications.IEEE Software40, 3 (2023), 4–8

    2023

  52. [60]

    David Lorge Parnas. 2010. Precise documentation: The key to better software. InThe future of software engineering. Springer, 125–148

    2010

  53. [61]

    Rolf-Helge Pfeiffer. 2020. What constitutes software? An empirical, descriptive study of artifacts. InProceedings of the 17th International Conference on Mining Software Repositories. 481–491

    2020

  54. [62]

    Marco Raglianti. 2022. Topology of the documentation landscape. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 297–299

    2022

  55. [63]

    Marco Raglianti, Csaba Nagy, Roberto Minelli, Bin Lin, and Michele Lanza. 2023. On the rise of modern software documentation (pearl/brave new idea). In37th European Conference on Object-Oriented Programming (ECOOP 2023). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 43–1

    2023

  56. [64]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends®in Information Retrieval3, 4 (2009), 333–389

    2009

  57. [65]

    Marcela Ruiz, Jin Yang Hu, and Fabiano Dalpiaz. 2023. Why don’t we trace? A study on the barriers to software traceability in practice.Requirements Engineering28, 4 (2023), 619–637

    2023

  58. [66]

    Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models.arXiv preprint arXiv:2310.10076(2023)

    2023 arXiv

  59. [67]

    SEART. 2025. GitHub Search. https://seart-ghs.si.usi.ch/ [Accessed 05-03-2025]

    2025

  60. [68]

    SEART. 2025. GitHub Search. https://github.com/seart-group/ghs [Accessed 05-03-2025]

    2025

  61. [69]

    Lakmal Silva, Michael Unterkalmsteiner, and Krzysztof Wnuk. 2023. Towards identifying and minimizing customer- facing documentation debt. In2023 ACM/IEEE International Conference on Technical Debt (TechDebt). IEEE, 72–81

    2023

  62. [70]

    Seoha Song, Junhyun Lee, and Hyeonmok Ko. 2025. Hansel: Output length controlling framework for large language models. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 25146–25154

    2025

  63. [71]

    Daniela Steidl, Benjamin Hummel, and Elmar Juergens. 2013. Quality analysis of source code comments. In2013 21st international conference on program comprehension (icpc). Ieee, 83–92

    2013

  64. [72]

    Wen Siang Tan, Markus Wagner, and Christoph Treude. 2024. Detecting outdated code element references in software repository documentation.Empirical Software Engineering29, 1 (2024), 5

    2024

  65. [73]

    Hassan, and Kenichi Matsumoto

    Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2017. An Empirical Compar- ison of Model Validation Techniques for Defect Prediction Models. 1 (2017)

    2017

  66. [74]

    Hassan, and Kenichi Matsumoto

    Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2018. The Impact of Automated Parameter Optimization for Defect Prediction Models. (2018). , Vol. 1, No. 1, Article . Publication date: September 2025. Evaluating the Use of LLMs for Documentati...

    2018

  67. [75]

    Fangchao Tian, Tianlu Wang, Peng Liang, Chong Wang, Arif Ali Khan, and Muhammad Ali Babar. 2021. The impact of traceability on software maintenance and evolution: A mapping study.Journal of Software: Evolution and Process33, 10 (2021), e2374

    2021

  68. [76]

    Gias Uddin and Martin P Robillard. 2015. How API documentation fails.Ieee software32, 4 (2015), 68–75

    2015

  69. [77]

    Esther Van der Linde, Daniel Kless, and Gustaf Neumann. 2020. Text Similarity Between Concepts Extracted from Source Code and Documentation. InAdvances in Artificial Intelligence. Springer, 72–85

    2020

  70. [78]

    Esther van der Linde, Frederik Schaefer, and Daniel Kless. 2021. Extracting and Comparing Concepts Emerging from Software Code, Documentation and Tests. InProceedings of the 2021 2nd International Workshop on AI in Requirements Engineering. ACM, 1–8

    2021

  71. [79]

    Alejandro Velasco and Jairo Aponte. 2020. Automated fine grained traceability links recovery between high level requirements and source code implementations.ParadigmPlus1, 2 (2020), 18–41

    2020

  72. [80]

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. 2024. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa.arXiv preprint arXiv:2406.17419(2024)

    2024 arXiv

  73. [81]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering.arXiv preprint arXiv:2502.06193(2025)

    2025 arXiv

  74. [82]

    Zezhou Yang, Sirong Chen, Cuiyun Gao, Zhenhao Li, Xing Hu, Kui Liu, and Xin Xia. 2025. An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  75. [83]

    Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an understanding of large language models in software engineering tasks.Empirical Software Engineering30, 2 (2025), 50

    2025

  76. [84]

    Hucheng Zhou, Jian-Guang Lou, Hongyu Zhang, Haibo Lin, Haoxiang Lin, and Tingting Qin. 2015. An empirical study on quality issues of production big data platform. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 17–26

    2015

  77. [85]

    Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs.arXiv preprint arXiv:2207.05987(2022)

    2022 arXiv

  78. [86]

    "" Process a single document request asynchronously . Prints status messages , builds the prompt , calls the LLM concurrently , and processes the response

    Thomas Zimmermann and Peter Weißgerber. 2004. Preprocessing CVS Data for Fine-Grained Analysis.. InMSR, Vol. 4. 2–6. A APPENDIX Below we provide additional details of our methodology. Figure 5 in the main body illustrates our prompt structure for documentation-to-code traceabi...

    2004

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.