{"id":"af5e06f4-ae66-48a3-8345-183f439b1ff1","arxiv_id":"2411.14971","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On legacy MUMPS code, LLM-generated line comments are rated almost as good as human-written ones; on IBM assembly they are rated low, and standard automated metrics do not predict human quality scores.","lead":"This paper tested whether large language models can write useful line-by-line comments for old software written in MUMPS and IBM mainframe assembly. The models did well on MUMPS, poorly on assembly, and no automated scoring method matched human judgment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ALC human ratings are too unreliable to support the paper's quantitative conclusions about ALC, including the claim that LLM comments are comparable to ground truth and that no automated metric correlates with quality.","rationale":"The reader identified the SME rating rubric as the weakest assumption, and I agree that it is the most load-bearing issue. The strongest claim splits into a MUMPS result and an ALC result plus a negative correlation finding. The MUMPS result is supported by moderate-to-good ICC values (0.65-0.89) and survives scrutiny. The ALC result and the ALC half of the correlation analysis rest on ratings with ICCs between 0.12 and 0.22, conventionally considered poor, and the reported usefulness CI [0.17, 0.37] around a point estimate of 0.12 is internally inconsistent. At this reliability level, ALC mean scores and source differences are not stable, and the null correlations could be attenuated by label noise. This is a correctness risk rather than a presentation issue because the paper presents these ALC numbers as baseline findings. The proposed checks are feasible and would settle whether the ALC conclusions are real: recompute ICCs per source, include reviewer as a covariate, and correct the impossible interval. These checks do not change the overall verdict: the paper should remain conditional, with the ALC-specific quantitative assertions gated on the reanalysis. The MUMPS evidence, the transparent discussion of inter-rater reliability, and the practical framing all count in the paper's favor, so no harsher action is warranted.","tokens_in":19841,"tokens_out":6180,"duration_ms":51146,"concrete_test":"Recalculate the ALC ICC values from the raw rating data, correcting the impossible usefulness confidence interval; then compute per-source ICC(2,k) for each LLM and for ground truth, and rerun the Tables IV-V correlation analysis on the subset of ALC comments rated by all five reviewers using a mixed-effects model with reviewer as a random effect. If per-source ICCs remain below ~0.5 and the mixed-model correlations stay near zero, the ALC conclusions should be reported only qualitatively; if ICCs or correlations change materially, the paper must revise its ALC-specific claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's ALC conclusions rest on SME ratings with very poor inter-rater reliability: Section IV.A reports ICC(2,k)=0.22 for hallucination, 0.16 for readability, 0.21 for completeness, and 0.12 for usefulness on ALC. The reported usefulness 95% CI [0.17, 0.37] is impossible for a point estimate of 0.12, indicating an error in the very statistic underlying the ALC claims. At these ICC levels, mean ratings in Figure 3 are largely a function of reviewer identity rather than comment quality, so the ALC portion of Hypothesis 1(a) and the ALC halves of Hypotheses 2 and 3 cannot be distinguished from label noise. Additionally, Tables IV and V pool all sources and treat ground truth as just another model; the paper does not report that reviewers were blind to source, so systematic source-level bias would masquerade as a quality signal. The paper acknowledges the ICC problem but still reports composite ALC means and correlations as findings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether LLMs can generate useful line-wise documentation for legacy code in MUMPS and IBM mainframe assembly (ALC). It proposes a prompting strategy in which existing comments are replaced by unique placeholders and the model returns JSON-structured comments, and it introduces a four-point SME rubric for hallucination, completeness, readability, and usefulness. Authors collected human ratings for four LLMs plus ground-truth comments on two real-world corpora (WorldVistA MUMPS and Walmart zFAM ALC), and then correlated those ratings with code-complexity, runtime-cost, and reference-based metrics. The main reported findings are that LLM-generated comments are rated comparable to or better than ground truth on MUMPS, that ALC comments are rated more poorly, and that no automated metric correlates strongly with human-rated quality.","tokens_in":1517,"tokens_out":2073,"duration_ms":94906,"significance":"If the empirical claims hold, the paper makes a useful contribution by supplying a concrete prompting strategy, two real-world legacy-code evaluation corpora, and a cautious negative result about automated documentation metrics. The authors are appropriately careful about multiple comparisons and the hypotheses are falsifiable. The MUMPS portion, in particular, is a plausible demonstration that LLM comment generation can help in at least one legacy language. However, the ALC conclusions rest on human ratings with very poor inter-rater reliability, the sampling and blinding of comments are not documented, and no data, code, or rating artifacts are provided. At this stage the contribution is strongest as a motivating case study and as evidence for the MUMPS setting, while the ALC-specific quantitative claims need substantial reworking.","major_comments":[{"comment":"The ALC inter-rater reliability statistics are internally inconsistent and too weak to support the ALC-specific conclusions. The usefulness ICC is reported as 0.12 with a 95% confidence interval of [0.17, 0.37], which excludes the point estimate and indicates an error in the printed statistic. More fundamentally, all four ALC ICC values (0.12-0.22) fall in the 'poor' range under the Koo & Li guideline cited by the authors, yet the same ratings are used to confirm Hypothesis 1(a) for ALC and to compute the ALC rows of Tables IV and V. Low inter-rater reliability means that mean ratings may reflect reviewer identity rather than comment quality, so the paper should either re-analyze the ALC data with reliability corrections or explicitly re-frame the ALC results as exploratory rather than confirmatory.","section":"Section IV.A"},{"comment":"The comment sampling and review protocol are under-specified. The paper states that only 127 of 7,097 ALC ground-truth comments and 544 of 21,698 ALC generated comments were rated, but it does not describe how these comments were selected, whether selection was random, or whether reviewers were blind to whether a comment was generated by an LLM or written by a human. Without this information, the mean ratings in Figure 3 and the correlation results in Tables IV and V may reflect selection bias or source-label bias. Please report the sampling procedure, the exact number of comments from each source and each reviewer, and the blinding protocol.","section":"Section III.C.1 and Section IV.A"},{"comment":"The correlation analysis is not specified precisely enough to be interpreted or reproduced. The paper reports Pearson correlations without stating the unit of analysis, the per-cell sample sizes, whether ground-truth comments are included in the pooled correlations, or how metrics such as processing time, cost, and retries are defined for human-written ground-truth comments. In addition, comments drawn from the same file or model are not independent, so uncorrected Pearson correlations may overstate significance. Please specify the exact construction of each correlation table, use cluster-robust or mixed-effect methods if pooling is retained, and, if possible, release the de-identified rating data to support the claims.","section":"Section IV.B, Section IV.C, Tables IV and V"},{"comment":"Hypothesis 1(b) is not operationalized precisely. The rubric in Table II uses a 4-point scale, but the hypothesis and the results section report ratings 'out of 10' (for example, 9.1/10 for GPT-4 Turbo and 6.53/10 for Llama3). The paper never states how the 4-point rubric is converted to a 10-point scale, nor does it justify the threshold of 7/10. Without an explicit conversion and threshold rationale, the absolute-score findings cannot be reproduced or compared across datasets.","section":"Section III.D and Section IV.A"}],"minor_comments":[{"comment":"The text says 'the comments generated from each of the twelve LLMs,' but only four models are used in the study; this should be corrected to 'four LLMs.'","section":"Section III.A.2"},{"comment":"The hypothesis statement contains the typo 'describd below' and should read 'described below.'","section":"Section III.D"},{"comment":"The word 'varibale' in the discussion of context-window sizes should be corrected to 'variable.'","section":"Section IV.A"},{"comment":"The caption contains a duplicated word: 'AND AND REFERENCE-BASED' should be 'AND REFERENCE-BASED.'","section":"Table III caption"},{"comment":"The displayed BLEU formula is not the standard BLEU formula; if a standard library was used, the formula should be corrected or replaced by a reference to the library.","section":"Table III"},{"comment":"The in-text reference 'Table IV-B' should be 'Table IV' to match the caption.","section":"Section IV.B"},{"comment":"The text references the 'blue, leftmost bar within each group,' but the figure as printed may not convey color; consider using distinct hatching or labels so the ground-truth bar is identifiable without color.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful core idea and a plausible MUMPS result, but the ALC half is not yet solid enough for the claims made. I would encourage the authors to focus the revision on transparency: full description of the sampling and blinding procedure, corrected and properly qualified ICC statistics, explicit correlation table construction, and release of the rating data. If those issues are addressed, the paper could become a valuable empirical contribution on LLM-generated documentation for legacy languages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper is a genuinely useful first look at LLM-generated line comments for two legacy languages that the documentation-generation literature has ignored, and the placeholder-prompting strategy is a real, reusable contribution. But the ALC quantitative claims currently sit on human ratings with poor inter-rater reliability, and one reported confidence interval is impossible; those numbers should be treated as provisional.\n\nWhat's new and good: the corpora (VistA MUMPS Incomplete Record Tracking and Walmart zFAM ALC) are realistic and non-trivial. The prompting method—replacing existing comments with placeholder IDs and requesting JSON output—cleanly prevents the LLM from rewriting code and preserves one-to-one mapping, which is a practical fix many will want to copy. The four-dimension rubric is sensible, and the authors are appropriately honest about multiple comparisons, telling readers not to over-read individual correlations.\n\nSoft spots, in order of importance. First, the ALC ratings: ICC(2,k) values of 0.12 to 0.22 mean the average ALC scores are largely reviewer identity, not comment quality. The paper acknowledges this but still reports composite means and correlations as findings. That is the load-bearing weakness for anything ALC-specific, including the claim that Hypothesis 1(a) holds for ALC. Second, the usefulness ICC is reported as 0.12 with 95% CI [0.17, 0.37]—impossible, since the interval excludes the point estimate. This looks like a typo, but it sits in the very statistic the ALC conclusions rest on and needs fixing. Third, the selection of comments sent to reviewers is not described; for ALC, only 127 of 7,097 ground-truth comments were rated, so we cannot assess representativeness. Fourth, no code, data, or rating artifacts are released, making the study non-reproducible. The lack of any blinding statement for raters is a related gap, though it would mostly affect claims of source-level differences rather than the negative correlation results.\n\nThe MUMPS arm is in better shape: ICCs are moderate-to-good, and the finding that MUMPS comments are readable and mostly hallucination-free is credible. The central negative result—that BLEU, ROUGE, CHRF, complexity, and cost metrics do not correlate strongly with human quality—is consistent with prior work and, if anything, conservative here, since noisy ALC labels would attenuate correlations.\n\nWho's this for? Teams building LLM-assisted modernization pipelines and anyone developing automated metrics for generated documentation. It deserves a serious referee, not a desk reject. I'd send it to review with a request for revision: fix the CI, add sampling details and a blinding protocol, and release whatever artifacts can be released. If the ALC arm can't be repaired, the paper should be reframed around MUMPS and the ALC results presented only as exploratory.","headline":"Valuable first run at LLM comment generation for MUMPS and ALC with a reusable prompting trick, but the ALC numbers rest on shaky human ratings and need serious revision.","tokens_in":20623,"tokens_out":4638,"would_cite":true,"duration_ms":42676,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that off-the-shelf LLMs can generate line-wise documentation for legacy MUMPS code at quality close to human-written comments, while for mainframe assembly the quality is lower; it also shows that no automated…","keywords":["large language models","legacy code modernization","MUMPS","mainframe assembly","code documentation generation","automated evaluation metrics","human evaluation","comment quality"],"falsifier":"A replication study on the same two corpora with a fresh panel of expert reviewers using the same rubric, in which inter-rater reliability on ALC exceeds 0.7 and at least one automated metric (for example cosine similarity on MUMPS) correlates above 0.7 with the averaged human scores, would contradict the paper's central negative claim that no automated metric strongly measures comment quality.","tokens_in":19644,"feed_emoji":"📝","tokens_out":6490,"duration_ms":55884,"temperature":0.7,"pith_summary":"The paper investigates whether mainstream LLMs can write trustworthy line-by-line documentation for legacy code written in MUMPS and IBM mainframe assembly language, and whether cheap automated metrics can substitute for expensive human review. Human subject-matter experts rated LLM-generated comments on MUMPS as comparable to, and in readability better than, the original human-written comments, while assembly-language comments were rated lower. On both languages, none of the tested automated metrics—code complexity, model runtime cost, BLEU, ROUGE, CHRF, readability scores, or embedding similarity—correlated strongly with human quality judgments, with the strongest correlation near 0.34. The authors conclude that LLM documentation is a viable aid for modernizing MUMPS code, that assembly code remains challenging for both humans and machines, and that no current automated metric can reliably screen or grade LLM-generated documentation.","feed_headline":"Human-level LLM docs for MUMPS, but metrics can't predict quality","feed_subtitle":"Expert raters score LLM comments near human-written ones on MUMPS, yet BLEU, ROUGE and cost metrics barely match those ratings.","key_machinery":"The comment-generation method replaces every human-written comment in a code chunk with a unique placeholder token (for example <BLOCK_COMMENT [id]> or <INLINE_COMMENT [id]>) and prompts the LLM to return a JSON object mapping each id to a new comment. This prevents the model from rewriting or inventing code, which was a failure mode in naive prompting, and preserves a one-to-one mapping between generated and ground-truth comments. The evaluation machinery is a four-dimension rubric with a 1–4 scale for hallucination, readability, completeness, and usefulness, applied by subject-matter experts, plus a battery of automated metrics (cyclomatic and Halstead complexity, maintainability index, MUMPS-specific pain-point counts, runtime cost, BLEU, ROUGE, CHRF, Flesch, Gunning Fog, and embedding cosine similarity) whose correlations with the human scores are then computed.","core_discovery":"The central finding is that, with a carefully designed prompting strategy, four mainstream LLMs produce line-wise comments for MUMPS that subject-matter experts rate close to or above the original developer-written comments across all four rubric dimensions (hallucination, readability, completeness, usefulness), while on ALC the absolute ratings are low and even the ground-truth human comments are scored inconsistently by the expert panel. The second major finding is that no automated measure—static complexity metrics, pain-point counts, processing time, monetary cost, BLEU, ROUGE, CHRF, Flesch, Gunning Fog, or cosine similarity between embeddings—achieves a strong correlation with any of the four human-rated dimensions. The highest significant correlation is cosine similarity versus usefulness on MUMPS, at about 0.34, which the authors describe as the most promising but still far from a reliable predictor. The paper therefore claims that current automated evaluation cannot tell an organization whether an LLM is producing good documentation for their legacy codebase.","pith_inferences":["Because the placeholder-and-JSON prompting strategy works on two very different legacy languages without code mutation, the same pattern could be used for other structured code-to-text tasks, such as generating function summaries or data-dictionary entries, with minimal per-language engineering.","The absence of a usable automated metric implies that organizations adopting LLM documentation for legacy code will need to budget for per-codebase human acceptance samples; this cost is not avoidable by using BLEU, ROUGE, or complexity thresholds.","If semantic or behavior-based evaluation (for example, checking whether a comment's claims match executed values or test outcomes) were developed, it might rescue automated screening where n-gram overlap and readability scores failed; this is an extrapolation the paper does not make."],"forward_implications":["MUMPS modernization teams can treat LLM line-comments as a viable first-pass documentation aid, since human experts rated them comparably to manually written comments on completeness and factualness.","For mainframe assembly, LLM-generated comments are not yet reliable enough for hands-off use, and even human raters disagree substantially about what a good comment is.","Automated metrics such as BLEU, ROUGE, CHRF, readability scores, complexity, and processing time cannot currently be used as quality gates for LLM documentation on legacy code.","The placeholder-based prompting strategy solves the twin failure modes of incomplete output and unwanted code mutation, preserving a clean one-to-one mapping to ground-truth comments.","Larger context windows did not improve line-level comment quality, so model choice for line-wise documentation should be driven by cost and factual accuracy rather than context size."],"supporting_citations":[{"why":"Establishes the baseline result that LLM-generated documentation can match developer-written documentation on modern code, which this paper extends to legacy languages.","marker":"[44]"},{"why":"Shows non-specialized LLMs outperform original documentation for inline and function-level comments, motivating the expectation tested here.","marker":"[45]"},{"why":"Documents that comment quality and metric reliability vary across codebases and that human evaluation is the reference, which is the exact issue the paper's metric correlations confront.","marker":"[20]"},{"why":"Previously found weak correlation between automated and human evaluation of code documentation quality, the same pattern this paper reports for legacy code.","marker":"[51]"},{"why":"Provides further evidence that common automated metrics do not align with human judgments of code summarization, supporting the paper's negative metric results.","marker":"[52]"},{"why":"Supplies the ROUGE reference-based metric used in the correlation analysis.","marker":"[55]"},{"why":"Supplies the CHRF metric used in the correlation analysis.","marker":"[56]"},{"why":"Supplies the BLEU metric whose weak correlation with human ratings is a central negative result.","marker":"[57]"},{"why":"Documents the lack of complex, realistic benchmark datasets for code summarization, which is why the authors built their own MUMPS and ALC corpora.","marker":"[18]"}],"fun_headline_variants":["LLMs match humans on MUMPS docs; metrics blind to it","On MUMPS, LLM docs rival humans; metrics stay clueless","LLMs write solid MUMPS docs, but automated scores fail","Legacy code docs: LLMs pass human bar, evaluation fails","LLMs meet human bar on MUMPS docs; no metric predicts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats the subject-matter-expert ratings from its own four-point rubric as the ground truth for comment quality, even though those ratings on the assembly-language dataset show poor rater agreement (ICCs between 0.12 and 0.22).","fun_headline_variants_meta":{"raw":{"variants":["LLMs match humans on MUMPS docs; metrics blind to it","On MUMPS, LLM docs rival humans; metrics stay clueless","LLMs write solid MUMPS docs, but automated scores fail","Legacy code docs: LLMs pass human bar, evaluation fails","LLMs meet human bar on MUMPS docs; no metric predicts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001194,"raw_usage":{"total_tokens":4936,"prompt_tokens":967,"completion_tokens":3969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":3873}},"tokens_in":583,"tokens_out":3969,"duration_ms":24603,"temperature":1.0,"reasoning_tokens":3873,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:39:34.592437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication study on the same two corpora with a fresh panel of expert reviewers using the same rubric, in which inter-rater reliability on ALC exceeds 0.7 and at least one automated metric (for example cosine similarity on MUMPS) correlates above 0.7 with the averaged human scores, would contradict the paper's central negative claim that no automated metric strongly measures comment quality.","supporting_citations":[{"cited_title":"Automatic code documentation generation using GPT-3,","cited_arxiv_id":null,"evidence_quote":"Establishes the baseline result that LLM-generated documentation can match developer-written documentation on modern code, which this paper extends to legacy languages."},{"cited_title":"A comparative analysis of large language models for code documentation generation,","cited_arxiv_id":null,"evidence_quote":"Shows non-specialized LLMs outperform original documentation for inline and function-level comments, motivating the expectation tested here."},{"cited_title":"A human study of comprehension and code summariza- tion,","cited_arxiv_id":null,"evidence_quote":"Documents that comment quality and metric reliability vary across codebases and that human evaluation is the reference, which is the exact issue the paper's metric correlations confront."},{"cited_title":"Correlating automated and human evaluation of code documentation generation quality,","cited_arxiv_id":null,"evidence_quote":"Previously found weak correlation between automated and human evaluation of code documentation quality, the same pattern this paper reports for legacy code."},{"cited_title":"On the evaluation of neural code summarization,","cited_arxiv_id":null,"evidence_quote":"Provides further evidence that common automated metrics do not align with human judgments of code summarization, supporting the paper's negative metric results."},{"cited_title":"ROUGE: A package for automatic evaluation of summaries,","cited_arxiv_id":null,"evidence_quote":"Supplies the ROUGE reference-based metric used in the correlation analysis."},{"cited_title":"chrF: Character n-gram F-score for automatic MT eval- uation,","cited_arxiv_id":null,"evidence_quote":"Supplies the CHRF metric used in the correlation analysis."},{"cited_title":"BLEU: A method for automatic evaluation of machine translation,","cited_arxiv_id":null,"evidence_quote":"Supplies the BLEU metric whose weak correlation with human ratings is a central negative result."},{"cited_title":"Recommendations for datasets for source code summarization,","cited_arxiv_id":null,"evidence_quote":"Documents the lack of complex, realistic benchmark datasets for code summarization, which is why the authors built their own MUMPS and ALC corpora."}],"review_version":1}