Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes that off-the-shelf LLMs can generate line-wise documentation for legacy MUMPS code at quality close to human-written comments, while for mainframe assembly the quality is lower; it also shows that no automated…

desk verdict Valuable first run at LLM comment generation for MUMPS and ALC with a reusable prompting trick, but the ALC numbers rest on shaky human ratings and need serious revision. read the letter →

arxiv 2411.14971 v1 pith:GP4XIAVN submitted 2024-11-22 cs.LG cs.SE

Colin Diggs , Michael Doyle , Amit Madan , Siggy Scott , Emily Escamilla , Jacob Zimmer , Naveed Nekoo , Paul Ursino
show 13 more authors
Michael Bartholf Zachary Robin Anand Patel Chris Glasz William Macke Paul Kirk Jasper Phillips Arun Sridharan Doug Wendt Scott Rosen Nitin Naik Justin F. Brunelle Samruddhi Thaker
This is my paper · ORCID
classification cs.LGcs.SE
keywords largelanguagemodelslegacycodemodernizationMUMPSmainframeassemblydocumentationgenerationautomatedevaluationmetricshumancommentquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates whether mainstream LLMs can write trustworthy line-by-line documentation for legacy code written in MUMPS and IBM mainframe assembly language, and whether cheap automated metrics can substitute for expensive human review. Human subject-matter experts rated LLM-generated comments on MUMPS as comparable to, and in readability better than, the original human-written comments, while assembly-language comments were rated lower. On both languages, none of the tested automated metrics—code complexity, model runtime cost, BLEU, ROUGE, CHRF, readability scores, or embedding similarity—correlated strongly with human quality judgments, with the strongest correlation near 0.34. The authors conclude that LLM documentation is a viable aid for modernizing MUMPS code, that assembly code remains challenging for both humans and machines, and that no current automated metric can reliably screen or grade LLM-generated documentation.

What carries the argument

The comment-generation method replaces every human-written comment in a code chunk with a unique placeholder token (for example <BLOCK_COMMENT [id]> or <INLINE_COMMENT [id]>) and prompts the LLM to return a JSON object mapping each id to a new comment. This prevents the model from rewriting or inventing code, which was a failure mode in naive prompting, and preserves a one-to-one mapping between generated and ground-truth comments. The evaluation machinery is a four-dimension rubric with a 1–4 scale for hallucination, readability, completeness, and usefulness, applied by subject-matter experts, plus a battery of automated metrics (cyclomatic and Halstead complexity, maintainability index, MUMPS-specific pain-point counts, runtime cost, BLEU, ROUGE, CHRF, Flesch, Gunning Fog, and embedding cosine similarity) whose correlations with the human scores are then computed.

What would settle it

A replication study on the same two corpora with a fresh panel of expert reviewers using the same rubric, in which inter-rater reliability on ALC exceeds 0.7 and at least one automated metric (for example cosine similarity on MUMPS) correlates above 0.7 with the averaged human scores, would contradict the paper's central negative claim that no automated metric strongly measures comment quality.

Watch

Extended reading notes

Core claim

The central finding is that, with a carefully designed prompting strategy, four mainstream LLMs produce line-wise comments for MUMPS that subject-matter experts rate close to or above the original developer-written comments across all four rubric dimensions (hallucination, readability, completeness, usefulness), while on ALC the absolute ratings are low and even the ground-truth human comments are scored inconsistently by the expert panel. The second major finding is that no automated measure—static complexity metrics, pain-point counts, processing time, monetary cost, BLEU, ROUGE, CHRF, Flesch, Gunning Fog, or cosine similarity between embeddings—achieves a strong correlation with any of the four human-rated dimensions. The highest significant correlation is cosine similarity versus usefulness on MUMPS, at about 0.34, which the authors describe as the most promising but still far from a reliable predictor. The paper therefore claims that current automated evaluation cannot tell an organization whether an LLM is producing good documentation for their legacy codebase.

Load-bearing premise

The paper treats the subject-matter-expert ratings from its own four-point rubric as the ground truth for comment quality, even though those ratings on the assembly-language dataset show poor rater agreement (ICCs between 0.12 and 0.22).

Editorial extensions

If this is right

  • MUMPS modernization teams can treat LLM line-comments as a viable first-pass documentation aid, since human experts rated them comparably to manually written comments on completeness and factualness.
  • For mainframe assembly, LLM-generated comments are not yet reliable enough for hands-off use, and even human raters disagree substantially about what a good comment is.
  • Automated metrics such as BLEU, ROUGE, CHRF, readability scores, complexity, and processing time cannot currently be used as quality gates for LLM documentation on legacy code.
  • The placeholder-based prompting strategy solves the twin failure modes of incomplete output and unwanted code mutation, preserving a clean one-to-one mapping to ground-truth comments.
  • Larger context windows did not improve line-level comment quality, so model choice for line-wise documentation should be driven by cost and factual accuracy rather than context size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the placeholder-and-JSON prompting strategy works on two very different legacy languages without code mutation, the same pattern could be used for other structured code-to-text tasks, such as generating function summaries or data-dictionary entries, with minimal per-language engineering.
  • The absence of a usable automated metric implies that organizations adopting LLM documentation for legacy code will need to budget for per-codebase human acceptance samples; this cost is not avoidable by using BLEU, ROUGE, or complexity thresholds.
  • If semantic or behavior-based evaluation (for example, checking whether a comment's claims match executed values or test outcomes) were developed, it might rescue automated screening where n-gram overlap and readability scores failed; this is an extrapolation the paper does not make.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper investigates whether LLMs can generate useful line-wise documentation for legacy code in MUMPS and IBM mainframe assembly (ALC). It proposes a prompting strategy in which existing comments are replaced by unique placeholders and the model returns JSON-structured comments, and it introduces a four-point SME rubric for hallucination, completeness, readability, and usefulness. Authors collected human ratings for four LLMs plus ground-truth comments on two real-world corpora (WorldVistA MUMPS and Walmart zFAM ALC), and then correlated those ratings with code-complexity, runtime-cost, and reference-based metrics. The main reported findings are that LLM-generated comments are rated comparable to or better than ground truth on MUMPS, that ALC comments are rated more poorly, and that no automated metric correlates strongly with human-rated quality.

Significance. If the empirical claims hold, the paper makes a useful contribution by supplying a concrete prompting strategy, two real-world legacy-code evaluation corpora, and a cautious negative result about automated documentation metrics. The authors are appropriately careful about multiple comparisons and the hypotheses are falsifiable. The MUMPS portion, in particular, is a plausible demonstration that LLM comment generation can help in at least one legacy language. However, the ALC conclusions rest on human ratings with very poor inter-rater reliability, the sampling and blinding of comments are not documented, and no data, code, or rating artifacts are provided. At this stage the contribution is strongest as a motivating case study and as evidence for the MUMPS setting, while the ALC-specific quantitative claims need substantial reworking.

major comments (4)
  1. [Section IV.A] The ALC inter-rater reliability statistics are internally inconsistent and too weak to support the ALC-specific conclusions. The usefulness ICC is reported as 0.12 with a 95% confidence interval of [0.17, 0.37], which excludes the point estimate and indicates an error in the printed statistic. More fundamentally, all four ALC ICC values (0.12-0.22) fall in the 'poor' range under the Koo & Li guideline cited by the authors, yet the same ratings are used to confirm Hypothesis 1(a) for ALC and to compute the ALC rows of Tables IV and V. Low inter-rater reliability means that mean ratings may reflect reviewer identity rather than comment quality, so the paper should either re-analyze the ALC data with reliability corrections or explicitly re-frame the ALC results as exploratory rather than confirmatory.
  2. [Section III.C.1 and Section IV.A] The comment sampling and review protocol are under-specified. The paper states that only 127 of 7,097 ALC ground-truth comments and 544 of 21,698 ALC generated comments were rated, but it does not describe how these comments were selected, whether selection was random, or whether reviewers were blind to whether a comment was generated by an LLM or written by a human. Without this information, the mean ratings in Figure 3 and the correlation results in Tables IV and V may reflect selection bias or source-label bias. Please report the sampling procedure, the exact number of comments from each source and each reviewer, and the blinding protocol.
  3. [Section IV.B, Section IV.C, Tables IV and V] The correlation analysis is not specified precisely enough to be interpreted or reproduced. The paper reports Pearson correlations without stating the unit of analysis, the per-cell sample sizes, whether ground-truth comments are included in the pooled correlations, or how metrics such as processing time, cost, and retries are defined for human-written ground-truth comments. In addition, comments drawn from the same file or model are not independent, so uncorrected Pearson correlations may overstate significance. Please specify the exact construction of each correlation table, use cluster-robust or mixed-effect methods if pooling is retained, and, if possible, release the de-identified rating data to support the claims.
  4. [Section III.D and Section IV.A] Hypothesis 1(b) is not operationalized precisely. The rubric in Table II uses a 4-point scale, but the hypothesis and the results section report ratings 'out of 10' (for example, 9.1/10 for GPT-4 Turbo and 6.53/10 for Llama3). The paper never states how the 4-point rubric is converted to a 10-point scale, nor does it justify the threshold of 7/10. Without an explicit conversion and threshold rationale, the absolute-score findings cannot be reproduced or compared across datasets.
minor comments (7)
  1. [Section III.A.2] The text says 'the comments generated from each of the twelve LLMs,' but only four models are used in the study; this should be corrected to 'four LLMs.'
  2. [Section III.D] The hypothesis statement contains the typo 'describd below' and should read 'described below.'
  3. [Section IV.A] The word 'varibale' in the discussion of context-window sizes should be corrected to 'variable.'
  4. [Table III caption] The caption contains a duplicated word: 'AND AND REFERENCE-BASED' should be 'AND REFERENCE-BASED.'
  5. [Table III] The displayed BLEU formula is not the standard BLEU formula; if a standard library was used, the formula should be corrected or replaced by a reference to the library.
  6. [Section IV.B] The in-text reference 'Table IV-B' should be 'Table IV' to match the caption.
  7. [Figure 3] The text references the 'blue, leftmost bar within each group,' but the figure as printed may not convey color; consider using distinct hatching or labels so the ground-truth bar is identifiable without color.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical evaluation whose conclusions are anchored in human SME ratings, not in fitted parameters or self-citation chains.

full rationale

The paper contains no derivation chain in which an output is defined in terms of its own input. The central claims are empirical: LLM-generated comments are compared against human subject-matter-expert ratings obtained with a rubric (Table II), and automated metrics are assessed by Pearson correlation against those same ratings (Tables IV and V). No coefficient is fitted to a subset of the human data and then 'predicted' as a result; the correlations are descriptive evaluations, not fitted predictors. The reference-based metrics (BLEU, ROUGE, CHRF, cosine similarity) are computed from the ground-truth comments and are never used to set a parameter that later reappears as a conclusion. The only self-citation in a substantive role is reference [49] (Macke & Doyle, which includes two of this paper's authors), used in a footnote to note that comments in input have little impact on code generation; this is a peripheral observation, not the load-bearing premise of any of the three hypotheses, and it reports an independent empirical result rather than an invoked uniqueness or ansatz. The acknowledged low ALC inter-rater reliability (ICC 0.12-0.22) is a serious measurement-quality limitation and could weaken the ALC conclusions, but it is not a circularity: noisy labels are still external evidence, not a restatement of the paper's inputs. The paper is therefore self-contained against its stated evaluation procedure; no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on human SME ratings, dataset representativeness, and hand-set experimental choices. There are no invented physical or mathematical entities; the parameters listed are procedural choices that could influence the empirical results.

free parameters (4)
  • Temperature = 0.7
    Set uniformly across Claude 3.0 Sonnet, Llama 3, Mixtral, and GPT-4 with no tuning. Comment quality and cost may depend on this choice.
  • Chunk merge threshold = half the model context window
    Greedy chunking stops merging when the chunk would exceed half the context window, reserving the other half for output. This hand-set threshold influences what context each model sees and could affect comment quality.
  • Hypothesis 1(b) absolute quality threshold = 7 out of 10 on average
    The confirmation of Hypothesis 1(b) for MUMPS but not ALC depends on this threshold; a different threshold would change the conclusion.
  • Retry policy = model re-run until all placeholders have comments
    Incomplete JSON outputs trigger retries; retries are included in processing time and cost metrics, affecting the cost and usefulness correlation results.
assumptions (4)
  • domain assumption SME ratings on a 4-point rubric are a valid proxy for comment quality.
    All conclusions about comment quality rest on four human reviewers for MUMPS and five for ALC. Low inter-rater reliability on ALC (ICC 0.12 to 0.22) weakens this premise.
  • domain assumption The Incomplete Record Tracking module is representative of the VistA MUMPS codebase.
    Section III.B.1 selects this module as representative of an average module based on comment and routine counts; if it is atypical, the MUMPS results do not generalize.
  • domain assumption The zFAM repository is representative of real-world ALC code.
    Section III.B.2 selects zFAM because it has many characteristics of real-world ALC codebases, but representativeness is asserted rather than demonstrated.
  • domain assumption Ground-truth human comments are an appropriate reference for reference-based metrics.
    The authors note in Section IV.A that original comments are not necessarily of good quality, yet BLEU, CHRF, ROUGE, and cosine similarity scores treat them as references.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation." pith.science (2026). https://pith.science/paper/GP4XIAVN

@misc{pith2026241114971,
  author       = {Pith},
  title        = {Pith review of: Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GP4XIAVN}},
  note         = {Machine review of arXiv:2411.14971}
}
read the original abstract

Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems.

Figures

Figures reproduced from arXiv: 2411.14971 by the authors.

Figure 1
Figure 1. The prompt template we used for line-wise comment generation. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Pre-processing procedure to replace existing in-line comments with [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Mean human evaluation scores for code comments generated by different LLMs (with 95% confidence intervals on the mean), for the MUMPS [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 3 citations worldwide. Full citation record

  1. Can LLMs Replace Humans During Code Chunking?

    cs.SE 2025-06 reject novelty 6.0 of 10

    LLM-generated partitions of legacy code yield documentation that LLM judges rate as up to 20% more factual and up to 10% more useful than documentation based on human expert partitions.

Reference graph

Works this paper leans on

65 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Information Technology: Agencies need to develop modernization plans for critical legacy sys- tems,

    U.S. Government Accountability Office , “Information Technology: Agencies need to develop modernization plans for critical legacy sys- tems,”

    work page

  2. [2]

    The legacy problem in government agencies: An exploratory study,

    A. Alexandrova, L. Rapanotti, and I. Horrocks, “The legacy problem in government agencies: An exploratory study,” in Proceedings of the 16th Annual International Conference on Digital Government Research . ACM, May 2015, pp. 150–159

    work page 2015

  3. [3]

    Federal IT Modernization Needs a Strategy and More Money,

    E. Egan, “Federal IT Modernization Needs a Strategy and More Money,” https://itif.org/publications/2022/05/31/ federal-it-modernization-needs-strategy-and-more-money/, 2022

    work page 2022

  4. [4]

    The future of GenAI will rocket fuel modernisation of core legacy systems,

    D. L. Giudice, “The future of GenAI will rocket fuel modernisation of core legacy systems,” Jul. 2024. [Online]. Available: https://www.ftadviser.com/platforms/2024/07/16/ the-future-of-genai-will-rocket-fuel-modernisation-of-core-legacy-systems/

    work page 2024

  5. [5]

    Ai-powered application rewrite: Revolutionizing legacy code transformation,

    S. Aulbach, “Ai-powered application rewrite: Revolutionizing legacy code transformation,” 2024. [Online]. Available: https://www2.deloitte.com/us/en/pages/consulting/articles/ ai-powered-application-rewrite-revolutionizing-legacy-code-transformation. html

    work page 2024

  6. [6]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto et al., “Evaluating large language models trained on code,” arXiv:2107.03374, Jul. 2021

    arXiv 2021

  7. [7]

    AI rewrites coding,

    S. Greengard, “AI rewrites coding,” Communications of the ACM , vol. 66, no. 4, pp. 12–14, Apr. 2023

    work page 2023

  8. [8]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-F oSE) . IEEE, May 2023, pp. 31–53

    work page 2023

Show all 65 references
  1. [9]

    Neural machine translation for code generation,

    D. KC and C. T. Morrison, “Neural machine translation for code generation,” arXiv:2305.13504, May 2023

    2023 arXiv

  2. [10]

    Large language models and simple, stupid bugs,

    K. Jesse, T. Ahmed, P. T. Devanbu, and E. Morgan, “Large language models and simple, stupid bugs,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, May 2023, pp. 563–575

    2023

  3. [11]

    Is Stack Overflow obsolete? An empirical study of the characteristics of ChatGPT answers to Stack Overflow questions,

    S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, “Is Stack Overflow obsolete? An empirical study of the characteristics of ChatGPT answers to Stack Overflow questions,” in Proceedings of the CHI Conference on Human Factors in Computing Systems . ACM, May 2024, pp. 1–17

    2024

  4. [12]

    Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,” in CHI Conference on Human Factors in Computing Systems Extended Abstracts . ACM, Apr. 2022, pp. 1–7

    2022

  5. [13]

    Better together? an evaluation of ai-supported code translation,

    J. D. Weisz, M. Muller, S. I. Ross, F. Martinez, S. Houde, M. Agarwal, K. Talamadupula, and J. T. Richards, “Better together? an evaluation of ai-supported code translation,” in 27th International Conference on Intelligent User Interfaces , 2022, pp. 369–391

    2022

  6. [14]

    Bridging Eras: Transforming Fortran legacies into Python with the power of large language models,

    R. Pietrini, M. Paolanti, and E. Frontoni, “Bridging Eras: Transforming Fortran legacies into Python with the power of large language models,” in 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI) . IEEE, Apr. 2024

    2024

  7. [15]

    A neural model for gener- ating natural language summaries of program subroutines,

    A. LeClair, S. Jiang, and C. McMillan, “A neural model for gener- ating natural language summaries of program subroutines,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, May 2019, pp. 795–806

    2019

  8. [16]

    Improved code summarization via a graph neural network,

    A. LeClair, S. Haque, L. Wu, and C. McMillan, “Improved code summarization via a graph neural network,” in Proceedings of the 28th International Conference on Program Comprehension. ACM, Jul. 2020, pp. 184–195

    2020

  9. [17]

    Improving automatic source code summarization via deep reinforcement learning,

    Y . Wan, Z. Zhao, M. Yang, G. Xu, H. Ying et al., “Improving automatic source code summarization via deep reinforcement learning,” in Pro- ceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, Sep. 2018, pp. 397–407

    2018

  10. [18]

    Recommendations for datasets for source code summarization,

    A. LeClair and C. McMillan, “Recommendations for datasets for source code summarization,” in Proceedings of the 2019 Conference of the North. Association for Computational Linguistics, 2019, pp. 3931– 3937

    2019

  11. [19]

    The evolution of a language standard: MUMPS in the 1980s,

    D. D. Sherertz, “The evolution of a language standard: MUMPS in the 1980s,” in Proceedings of the ACM 1980 annual conference . ACM Press, 1980, pp. 101–104

    1980

  12. [20]

    A human study of comprehension and code summariza- tion,

    S. Stapleton, Y . Gambhir, A. LeClair, Z. Eberhart, W. Weimer, K. Leach, and Y . Huang, “A human study of comprehension and code summariza- tion,” in Proceedings of the 28th International Conference on Program Comprehension. ACM, Jul. 2020, pp. 2–13

    2020

  13. [21]

    Information technology: Agencies need to continue addressing critical legacy systems,

    U.S. Government Accountability Office , “Information technology: Agencies need to continue addressing critical legacy systems,”

  14. [22]

    Information Technology: Federal agencies need to address aging legacy systems,

    ——, “Information Technology: Federal agencies need to address aging legacy systems,”

  15. [23]

    Information technology: IRS needs to complete modernization plans and fully address cloud comput- ing requirements,

    U.S. Government Accountability Office, “Information technology: IRS needs to complete modernization plans and fully address cloud comput- ing requirements,”

  16. [24]

    Chapter 12 - Veterans Health Admin- istration’s VistA MUMPS modernization pilot**© 2010. The Software Revolution, Inc. all rights reserved

    P. H. Newcomb and R. Couch, “Chapter 12 - Veterans Health Admin- istration’s VistA MUMPS modernization pilot**© 2010. The Software Revolution, Inc. all rights reserved.” in Information Systems Transfor- mation, W. M. Ulrich and P. H. Newcomb, Eds. Morgan Kaufmann, 2010, pp. 301–345

    2010

  17. [25]

    Open challenges in incremental coverage of legacy software languages,

    V . Zaytsev, “Open challenges in incremental coverage of legacy software languages,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Programming Experience . ACM, Oct. 2017, pp. 1–6

    2017

  18. [26]

    Verified code transpilation with LLMs,

    S. Bhatia, J. Qiu, N. Hasabnis, S. A. Seshia, and A. Cheung, “Verified code transpilation with LLMs,” arXiv:2406.03003, Jun. 2024

    2024 arXiv

  19. [27]

    Lost in translation: A study of bugs introduced by large language models while translating code,

    R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi et al. , “Lost in translation: A study of bugs introduced by large language models while translating code,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . ACM, Apr. 2024, pp. 1–13

    2024

  20. [28]

    Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks,

    Z. Wu, L. Qiu, A. Ross, E. Aky ¨urek, B. Chen et al. , “Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks,” arXiv:2307.02477, Mar. 2024

    2024 arXiv

  21. [29]

    Neutron: an attention-based neural decompiler,

    R. Liang, Y . Cao, P. Hu, and K. Chen, “Neutron: an attention-based neural decompiler,” Cybersecurity, vol. 4, no. 1, p. 5, Dec. 2021

    2021

  22. [30]

    A Neural-based program decompiler,

    C. Fu, H. Chen, H. Liu, X. Chen, Y . Tian et al. , “A Neural-based program decompiler,” Advances in Neural Information Processing Systems, Jun. 2019. [Online]. Available: http://arxiv.org/abs/1906.12029

    2019 arXiv

  23. [31]

    Towards neural decompilation,

    O. Katz, Y . Olshaker, Y . Goldberg, and E. Yahav, “Towards neural decompilation,” arXiv:1905.08325, May 2019

    1905 arXiv

  24. [32]

    Automatic source code summa- rization of context for Java methods,

    P. W. McBurney and C. McMillan, “Automatic source code summa- rization of context for Java methods,” IEEE Transactions on Software Engineering, vol. 42, no. 2, pp. 103–119, Feb. 2016

    2016

  25. [33]

    Automatic generation of natural language summaries for Java classes,

    L. Moreno, J. Aponte, G. Sridhara, A. Marcus, L. Pollock, and K. Vijay- Shanker, “Automatic generation of natural language summaries for Java classes,” in 2013 21st International Conference on Program Compre- hension (ICPC) . IEEE, May 2013, pp. 23–32

    2013

  26. [34]

    Towards automatically generating summary comments for Java meth- ods,

    G. Sridhara, E. Hill, D. Muppaneni, L. Pollock, and K. Vijay-Shanker, “Towards automatically generating summary comments for Java meth- ods,” in Proceedings of the IEEE/ACM international conference on Automated software engineering . ACM, Sep. 2010, pp. 43–52

    2010

  27. [35]

    CloCom: Mining existing source code for automatic comment generation,

    E. Wong, Taiyue Liu, and L. Tan, “CloCom: Mining existing source code for automatic comment generation,” in 2015 IEEE 22nd Interna- tional Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, Mar. 2015, pp. 380–389

    2015

  28. [36]

    A survey of automatic source code summarization,

    C. Zhang, J. Wang, Q. Zhou, T. Xu, K. Tang, H. Gui, and F. Liu, “A survey of automatic source code summarization,” Symmetry, vol. 14, no. 3, p. 471, Feb. 2022

    2022

  29. [37]

    Automatic code summarization: A systematic literature review,

    Y . Zhu and M. Pan, “Automatic code summarization: A systematic literature review,” arXiv:1909.04352, Oct. 2019

    1909 arXiv

  30. [38]

    Improving code summarization with block-wise abstract syntax tree splitting,

    C. Lin, Z. Ouyang, J. Zhuang, J. Chen, H. Li, and R. Wu, “Improving code summarization with block-wise abstract syntax tree splitting,” in 2021 IEEE/ACM 29th International Conference on Program Compre- hension (ICPC) . IEEE, May 2021, pp. 184–195

    2021

  31. [39]

    Deep code comment generation with hybrid lexical and syntactical information,

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation with hybrid lexical and syntactical information,” Empirical Software Engineering, vol. 25, no. 3, pp. 2179–2217, May 2020

    2020

  32. [40]

    Summarizing source code using a neural attention model,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) . Berlin, Germany: Association for Computational Li...

    2016

  33. [41]

    Code structure–Guided transformer for source code summarization,

    S. Gao, C. Gao, Y . He, J. Zeng, L. Nie et al. , “Code structure–Guided transformer for source code summarization,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 1, pp. 1–32, Jan. 2023

    2023

  34. [42]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora et al., “On the opportunities and risks of foundation models,” arXiv:2108.07258, Jul. 2022

    2022 arXiv

  35. [43]

    Assemble foundation models for automatic code summarization,

    J. Gu, P. Salza, and H. C. Gall, “Assemble foundation models for automatic code summarization,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, Mar. 2022, pp. 935–946

    2022

  36. [44]

    Automatic code documentation generation using GPT-3,

    J. Y . Khan and G. Uddin, “Automatic code documentation generation using GPT-3,” Sep. 2022

    2022

  37. [45]

    A comparative analysis of large language models for code documentation generation,

    S. S. Dvivedi, V . Vijay, S. L. R. Pujari, S. Lodh, and D. Kumar, “A comparative analysis of large language models for code documentation generation,” in Proceedings of the 1st ACM International Conference on AI-Powered Software. ACM, Jul. 2024, pp. 65–73

    2024

  38. [46]

    Comparing code explanations created by students and large language models,

    J. Leinonen, P. Denny, S. MacNeil, S. Sarsa, S. Bernstein, J. Kim, A. Tran, and A. Hellas, “Comparing code explanations created by students and large language models,” in Proceedings of the 2023 Con- ference on Innovation and Technology in Computer Science Education V. 1 . ACM...

    2023

  39. [47]

    Using an LLM to help with code understanding,

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an LLM to help with code understanding,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . ACM, Apr. 2024, pp. 1–13

    2024

  40. [48]

    Code summarization: Do transformers really understand code?

    A. N. Sontakke, M. Patwardhan, L. Vig, R. K. Medicherla, R. Naik, and G. Shroff, “Code summarization: Do transformers really understand code?” in Deep Learning for Code Workshop , 2022. [Online]. Available: https://openreview.net/forum?id=rI5ll2 -1Zc

    2022

  41. [49]

    Testing the effect of code documentation on large language model code understanding,

    W. Macke and M. Doyle, “Testing the effect of code documentation on large language model code understanding,” in Findings of the Association for Computational Linguistics: NAACL 2024 . Association for Computational Linguistics, 2024, pp. 1044–1050

    2024

  42. [50]

    Evaluating software documentation quality,

    H. Tang and S. Nadi, “Evaluating software documentation quality,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, May 2023, pp. 67–78

    2023

  43. [51]

    Correlating automated and human evaluation of code documentation generation quality,

    X. Hu, Q. Chen, H. Wang, X. Xia, D. Lo, and T. Zimmermann, “Correlating automated and human evaluation of code documentation generation quality,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 4, pp. 1–28, Oct. 2022

    2022

  44. [52]

    On the evaluation of neural code summarization,

    E. Shi, Y . Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” in Pro- ceedings of the 44th International Conference on Software Engineering . ACM, May 2022, pp. 1597–1608

    2022

  45. [53]

    ”We Need Structured Output

    M. X. Liu, F. Liu, A. J. Fiannaca, T. Koo, L. Dixon, M. Terry, and C. J. Cai, “”We Need Structured Output”: Towards user-centered constraints on large language model output,” in Extended Abstracts of the CHI Conference on Human Factors in Computing Systems , May 2024, pp. 1–9

    2024

  46. [54]

    Label Studio: Data labeling software,

    M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, “Label Studio: Data labeling software,” 2020-2022, open source software available from https://github.com/heartexlabs/label-studio. [Online]. Available: https://github.com/heartexlabs/label-studio

    2020

  47. [55]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out . Association for Computational Linguistics, 2004, pp. 74–81

    2004

  48. [56]

    chrF: Character n-gram F-score for automatic MT eval- uation,

    M. Popovi ´c, “chrF: Character n-gram F-score for automatic MT eval- uation,” in Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, 2015, pp. 392– 395

    2015

  49. [57]

    BLEU: A method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02. Association for Computational Linguistics, 2001, p. 311

    2001

  50. [58]

    A guideline of selecting and reporting intraclass correlation coefficients for reliability research,

    T. K. Koo and M. Y . Li, “A guideline of selecting and reporting intraclass correlation coefficients for reliability research,” Journal of Chiropractic Medicine, vol. 15, no. 2, pp. 155–163, Jun. 2016

    2016

  51. [59]

    Practitioners’ expectations on automated code comment generation,

    X. Hu, X. Xia, D. Lo, Z. Wan, Q. Chen, and T. Zimmermann, “Practitioners’ expectations on automated code comment generation,” in Proceedings of the 44th International Conference on Software Engi- neering. ACM, May 2022, pp. 1693–1705

    2022

  52. [60]

    Leveraging large language models for NLG evaluation: Advances and challenges,

    Z. Li, X. Xu, T. Shen, C. Xu, J.-C. Gu et al. , “Leveraging large language models for NLG evaluation: Advances and challenges,” arXiv:2401.07103, Jun. 2024

    2024 arXiv

  53. [61]

    Ibm unveils watsonx generative AI capabilities to accelerate mainframe application modernization,

    IBM, “Ibm unveils watsonx generative AI capabilities to accelerate mainframe application modernization,”

  54. [62]

    Parameter-efficient fine- tuning for large models: A comprehensive survey,

    Z. Han, C. Gao, J. Liu, S. Q. Zhang et al. , “Parameter-efficient fine- tuning for large models: A comprehensive survey,” arXiv:2403.14608, 2024

    2024 arXiv

  55. [63]

    Semantic similarity metrics for evaluating source code summarization,

    S. Haque, Z. Eberhart, A. Bansal, and C. McMillan, “Semantic similarity metrics for evaluating source code summarization,” inProceedings of the 30th IEEE/ACM International Conference on Program Comprehension . ACM, May 2022, pp. 36–47

    2022

  56. [64]

    Evaluating code summarization techniques: A new metric and an empirical charac- terization,

    A. Mastropaolo, M. Ciniselli, M. D. Penta, and G. Bavota, “Evaluating code summarization techniques: A new metric and an empirical charac- terization,” arXiv:2312.15475, Dec. 2023

    2023 arXiv

  57. [2023]

    Available: https://newsroom.ibm.com/ 2023-08-22-IBM-Unveils-watsonx-Generative-AI-Capabilities-to-Accelerate-Mainframe-Application-Modernization

    [Online]. Available: https://newsroom.ibm.com/ 2023-08-22-IBM-Unveils-watsonx-Generative-AI-Capabilities-to-Accelerate-Mainframe-Application-Modernization

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.