Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

The paper argues that mutation-based testing—changing a program's behavior and checking whether the model's summary changes—can expose when large language models describe what code looks like it should do rather than what it actually does.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:05 UTC pith:YSZUZ4VB

load-bearing objection Useful mutation-based evaluation framework for LLM code summaries, but the headline accuracy rates depend on an unverified assumption that every mutant actually changes behavior. the 5 major comments →

arxiv 2602.17838 v2 pith:YSZUZ4VB submitted 2026-02-19 cs.SE

Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code

Lara Khatib , Michael Pu , Bogdan Vasilescu , Meiyappan Nagappan This is my paper
classification cs.SE
keywords mutation analysiscode summarizationlarge language modelsprogram behavior vs intentevaluation methodologyGPT-4summary accuracy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish a way to test whether LLM-generated code summaries are actually grounded in program behavior. The method: summarize the code, make a small behavior-changing edit (a mutation), summarize the edited code, and ask whether a human can spot the edit by comparing the two summaries. Across 624 mutation-summary comparisons, the authors report that accuracy falls sharply as code grows more complex—from 76.5% for single functions to 17.3% for multithreaded systems—and that GPT-4 misses roughly half of mutations in human-written programs while GPT-5.2 misses about 15%. If correct, this gives developers and researchers a repeatable way to check whether a summary follows the logic of the code rather than surface patterns.

Core claim

The central claim is that mutation analysis provides a direct probe for whether a summary tracks behavior: if a change that alters what the program does is not reflected in the corresponding summary, that summary is unreliable regardless of how fluent it reads. The authors find a consistent failure mode in which models describe the intended or canonical algorithm—merge sort, heap operations, union-find—even after the mutation breaks that behavior. They operationalize this with a positive/negative classification: positive if the behavioral change can be identified from the two summaries, negative if not. The strongest empirical result is a steep complexity gradient, with accuracy falling mono

What carries the argument

The load-bearing object is the mutation—a small, behavior-changing edit (statement, value, or decision) placed at a chosen location—used as a probe. The evaluation pipeline is: (1) generate mutants from original code; (2) generate a fresh, temperature-0 summary for the original and each mutant with the same prompt; (3) have human evaluators decide whether the change can be identified by comparing summaries. The mutation is doing the work of turning 'does the summary reflect the code' into a concrete yes/no question: if the summary is keyed to actual behavior, a behavior change must show up in the summary.

Load-bearing premise

The load-bearing premise is that a human's ability to detect the mutation by comparing two summaries is a faithful measure of summary accuracy—if a summary is accurate but too coarse to mention a mutated detail, the method counts it as a failure, and that conflation would change every reported rate.

What would settle it

Take the 150 human-written-program mutation pairs and have a fresh panel of independent annotators classify them with the same rubric without seeing the authors' labels. If their agreement with the published labels is low (for instance, Cohen's kappa below 0.6), the accuracy rates are not stable ground truth. A second falsifier: if expert human-written summaries of the same mutated programs fail the two-summary comparison as often as the model summaries, then the test is measuring summary detail, not correctness.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If mutation-based evaluation is sound, it offers a repeatable stress test for any code summarizer, independent of reference summaries or n-gram overlap metrics.
  • The reported complexity gradient implies real-world, multi-component codebases are exactly where LLM summaries are least trustworthy.
  • The persistent failure mode—describing canonical intent over mutated behavior—implies that fluent summaries can actively mislead code review and documentation use.
  • The GPT-4 to GPT-5.2 comparison (49.3% to 85.3% on human-written single-function programs) suggests the method can measure generational progress under identical conditions.
  • Because the human-written programs are mostly single functions, the 85.3% figure should be read as a best-case; the authors expect performance to degrade on larger, interconnected code.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be automated by using a second large language model as the comparator, replacing human judgment, which would make it scalable to entire repositories—though this would require validating that the model judge agrees with human labels.
  • The same protocol could become a contamination-resistant benchmark for code summarization by standardizing mutations across programming languages and measuring detection rates, an alternative to leaderboard-style benchmarks.
  • A testable extension: prompt the model to explicitly identify the mutation after summarizing; if the model can name the change but does not incorporate it into the summary, that would separate extraction failure from verbalization failure.
  • The finding that hard-coded string literals can anchor a model's interpretation suggests a specific intervention: perturb comments, names, and literals to see whether summaries track semantics or lexical cues.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a mutation-based methodology for evaluating LLM-generated code summaries. For each code sample, the authors generate an LLM summary, inject a targeted mutation, generate a summary for the mutated code, and then ask human annotators whether the mutation can be identified by comparing the two summaries. A positive classification means the mutation is detectable from the summaries; a negative means it is not. The methodology is applied in three experiments: (1) 324 mutations across 12 synthetic programs spanning four complexity levels, reporting a decline in detection from 76.5% (single function) to 17.3% (multi-threaded); (2) 150 mutations across 50 human-written LBPP programs, reporting 49.3% detection for GPT-4; and (3) the same 150 LBPP mutations evaluated with GPT-5.2, reporting 85.3% detection. The paper concludes that LLM summaries often track intended algorithmic patterns rather than actual mutated behavior, and argues that mutation analysis is a practical framework for comparing models and configurations.

Significance. The proposed methodology is a useful departure from n-gram-based summary metrics: it creates a behavioral probe of whether summaries track code changes, and the RQ1 experiment includes dual annotation with high inter-rater agreement (96.6%, Cohen's kappa = 0.928). The qualitative failure-mode analysis (missing details vs. hallucinated intent) is instructive, and the replication package is a tangible asset. If the validity concerns are resolved, the approach could provide a repeatable evaluation tool for code summarization. However, the three main quantitative claims suffer from load-bearing validity problems: the mutated programs are never executed to confirm that the mutations change behavior, the dependent variable is human detectability of a mutation from paired summaries rather than summary correctness, and the RQ2/RQ3 headline numbers rest on a single unblinded annotator. These issues are substantial but addressable, so the contribution is promising rather than invalid.

major comments (5)
  1. [Section 4, Section 6.2] The dependent variable presupposes that every injected mutation changes observable behavior. Section 6.2 states that 'what matters for our analysis is that the mutation changes the behavior of the code in a meaningful way,' but no execution, test, or formal equivalence check is reported. If any mutant is equivalent (same I/O behavior for all inputs), the correct summary should not change, yet the paper counts an unchanged summary as a negative. This threatens the denominator of every reported rate (76.5%, 49.3%, 85.3%, Table 1). Please run the original and mutated programs on a test suite or otherwise verify behavioral change, exclude or quantify equivalent mutants, and re-analyze the data.
  2. [Section 4, Section 5.1, Abstract] The paper equates 'mutation detectable from paired summaries' with 'summary accuracy.' A summary can be accurate but too coarse to mention a mutated detail; the paper's own first failure mode, 'summary is too abstract it omits the mutated logic entirely,' is treated as a failure without arguing that a correct summary must encode every behavioral change at that granularity. Conversely, two summaries can differ for reasons unrelated to mutation. The headline percentage is therefore better described as mutation sensitivity, not summary correctness. Please either reframe the claims as sensitivity to behavioral perturbations or add an independent human assessment of whether the mutated summary is actually incorrect.
  3. [Section 6.1, Section 5.2, Section 5.3] RQ2 and RQ3 classifications were performed by a single author, using the same criteria as RQ1 but on new data and on a second model's outputs. The RQ1 kappa of 0.928 does not establish reliability for the LBPP summaries or for GPT-5.2, and the annotator was not reported to be blinded to which model produced each summary. The 49.3% and 85.3% results are central to the paper. Please add a second annotator on a random subset (or all of RQ2/RQ3), report agreement, and blind the model identity during annotation.
  4. [Section 5.1] The chi-square test for complexity (chi-square(3)=69.04, p<0.001, V=0.462) treats 324 mutations as independent, but the mutations are nested within 12 programs (27 per program). Program-level correlation can inflate significance and understate uncertainty, and the same issue applies to the mutation-type and location tests and to the Mann-Whitney test on LOC. Please use a mixed-effects model, cluster-robust inference, or a program-level permutation test, or at least report intraclass correlation, to support the complexity and size conclusions.
  5. [Section 4.2] The definition of a negative outcome conflates two distinct cases: (a) the mutated summary is inaccurate because it describes original behavior, and (b) both summaries are accurate but too high-level to expose the mutation. The text says, 'if neither summaries explicitly described the mutated behaviour. That is, if the original and mutated summary both accurately describe the original and mutated code.' The second clause does not follow from the first: a summary that fails to describe the mutated behavior is not accurately describing the mutated code. Clarify the definition and separate these two cases in the analysis, since they have different implications.
minor comments (5)
  1. [Listing 4 caption] The caption says 'The < sign on line 3 was changed to > on line 8,' but the code shows the original uses '>' and the mutant uses '<'. The direction is reversed.
  2. [Listing 5 caption] The caption says 'sample02/desc/b2 from sample01' and the text describes merge sort, but the code shown is a union-find 'find' function. The sample identifier and code snippet do not match.
  3. [Section 1, Listing 1 caption] The caption says the sample is 'val_e_2 from sample05,' but the function shown is 'get_min' from a heap class; make the naming consistent with the sample identifiers used elsewhere.
  4. [Section 6.2] The sentence 'the point here is not to model typical bugs, it is to make sure the code's behavior actually changes' is an assertion, not a verification. It would be stronger to cite the execution results requested in the major comment.
  5. [Section 5.2] The phrase 'LBPP was explicitly designed to be uncontaminated problems and solutions were verified not to overlap' is awkward; consider rewording and citing the specific verification procedure.

Circularity Check

0 steps flagged

No circular derivation: the paper reports an empirical measurement; its operationalization and unverified mutation-behavior assumption are validity threats, not circularity.

full rationale

The paper does not derive predictions from equations or fit parameters. It defines an outcome (whether a human evaluator can identify the mutation from paired summaries), applies that definition to 624 mutation–summary pairs, and reports counts and proportions. The percentages (76.5%, 49.3%, 85.3%) are summaries of those human labels, not quantities derived from the labels by construction in a way that conceals an input. The connection between 'mutation not detected in summaries' and 'summary inaccurate' is an explicitly stated operationalization (Section 4, 'Compare Summaries'), and the paper even acknowledges the two failure modes (too abstract vs. hallucinated intent). The main correctness risk—that mutants may not actually change observable behavior (Section 6.2 asserts this without executing the code)—would threaten the validity of the ground truth, but it is not a circular step: the result does not reduce to an assumption of the paper by definition. The only self-citation ([26] in support of the prompt) is not load-bearing; the same point is supported by independent citations [13,31]. Therefore no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no mathematical free parameters or invented entities. Its load-bearing assumptions are about the validity of the human classification process, the behavior-changing nature of the mutations, and the determinism of LLM outputs. These assumptions are reasonable but not independently verified in the text.

axioms (4)
  • domain assumption Fresh sessions with temperature 0 yield deterministic, context-free summaries.
    Section 4 step (2). If LLM outputs are nondeterministic or carry context between sessions, observed summary differences could be noise rather than responses to mutations.
  • domain assumption Manually constructed mutations are behavior-changing and representative of meaningful code edits.
    Section 6.2. If many mutations do not actually change behavior, or if they are unrepresentative, the detection rates are not about behavior tracking.
  • domain assumption A human comparing the original and mutated summaries can reliably identify whether the behavioral change is reflected.
    Section 4 step (3). The entire accuracy measure depends on this judgment; a summary can be accurate but too coarse to mention the mutated detail.
  • domain assumption The LBPP dataset is uncontaminated and human-written.
    Section 5.2 cites [22]. If LBPP problems overlap with model training data, the RQ2/RQ3 comparisons could be inflated by memorization.

pith-pipeline@v1.3.0-alltime-deepseek · 13447 in / 8847 out tokens · 82659 ms · 2026-08-02T22:05:32.176700+00:00 · methodology

0 comments
read the original abstract

As developers increasingly rely on LLM-generated code summaries for documentation, testing, and review, it is important to study whether these summaries accurately reflect what the program actually does. LLMs often produce confident descriptions of what the code looks like it should do (intent), while missing subtle edge cases or logic changes that define what it actually does (behavior). We present a mutation-based evaluation methodology that directly tests whether a summary truly matches the code's logic. Our approach generates a summary, injects a targeted mutation into the code, and checks if the LLM updates its summary to reflect the new behavior. We validate it through three experiments totalling 624 mutation-summary evaluations across 62 programs. First, on 12 controlled synthetic programs with 324 mutations varying in type (statement, value, decision) and location (beginning, middle, end). We find that summary accuracy decreases sharply with complexity from 76.5% for single functions to 17.3% for multi-threaded systems, while mutation type and location exhibit weaker effects. Second, testing 150 mutated samples on 50 human-written programs from the Less Basic Python Problems (LBPP) dataset confirms the same failure patterns persist as models often describe algorithmic intent rather than actual mutated behavior with a summary accuracy rate of 49.3%. Furthermore, while a comparison between GPT-4 and GPT-5.2 shows a substantial performance leap (from 49.3% to 85.3%) and an improved ability to identify mutations as "bugs", both models continue to struggle with distinguishing implementation details from standard algorithmic patterns. This work establishes mutation analysis as a systematic approach for assessing whether LLM-generated summaries reflect program behavior rather than superficial textual patterns.

Figures

Figures reproduced from arXiv: 2602.17838 by Bogdan Vasilescu, Lara Khatib, Meiyappan Nagappan, Michael Pu.

Figure 1
Figure 1. Figure 1: Overview of process. code or not. We can only see that it can summarize code. Therefore, we stick to code summarization as the topic for this paper 3 Research Questions We validate our mutation-based evaluation methodology through three questions: • RQ1: How do complexity, program size, mutation type, and mutation location influence the model’s summarization performance? • RQ2: Are the same limitations see… view at source ↗
Figure 3
Figure 3. Figure 3: Results across the complexity categories. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results across the different mutation types, grouped [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results across the different mutation locations, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey

    cs.SE 2026-05 accept novelty 4.0

    A systematic survey of 93 studies that maps the bidirectional relationship between metamorphic testing and LLMs, proposing a taxonomy for MT applied to LLMs and LLMs applied to MT.

Reference graph

Works this paper leans on

39 extracted references · 11 linked inside Pith · cited by 1 Pith paper

  1. [1]

    [n. d.]. https://github.com/features/copilot

  2. [2]

    [n. d.]. Artificial Intelligence - the state of developer ecosystem in 2023. https: //www.jetbrains.com/lp/devecosystem-2023/ai/

    2023

  3. [3]

    Introducing ChatGPT

    2022. Introducing ChatGPT. https://openai.com/index/chatgpt

    2022

  4. [4]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summa- rization). InProceedings of the IEEE/ACM 46th international conference on software engineering. 1–13

    2024

  5. [5]

    James H Andrews, Lionel C Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering32, 8 (2006), 608–624

    2006

  6. [6]

    Anthropic. 2025. Tracing the thoughts of a large language model. https://www. anthropic.com/research/tracing-thoughts-language-model. Accessed: 2025-05- 22

    2025

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  9. [9]

    José Aldo Silva da Costa, Rohit Gheyi, Fernando Castor, Pablo Roberto Fernandes de Oliveira, Márcio Ribeiro, and Baldoino Fonseca. 2023. Seeing confusion through a new lens: on the impact of atoms of confusion on novices’ code comprehension.Empirical Software Engineering28, 4 (2023), 81

    2023

  10. [10]

    Benedito De Oliveira, Márcio Ribeiro, José Aldo Silva Da Costa, Rohit Gheyi, Guilherme Amaral, Rafael de Mello, Anderson Oliveira, Alessandro Garcia, Ro- drigo Bonifácio, and Baldoino Fonseca. 2020. Atoms of confusion: The eyes do not lie. InProceedings of the XXXIV Brazilian Symposium on Software Engineering. 243–252

    2020

  11. [11]

    Dan Gopstein, Jake Iannacone, Yu Yan, Lois DeLong, Yanyan Zhuang, Martin K-C Yeh, and Justin Cappos. 2017. Understanding misunderstandings in source code. InProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 129–139

    2017

  12. [12]

    Sabaat Haroon, Ahmad Faraz Khan, Ahmad Humayun, Waris Gill, Abdul Haddi Amjad, Ali R Butt, Mohammad Taha Khan, and Muhammad Ali Gulzar. 2025. How Accurately Do Large Language Models Understand Code?arXiv preprint arXiv:2504.04372(2025)

    arXiv 2025

  13. [13]

    Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing code explanations created by students and large language models. InProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1. 124–130

    2023

  14. [14]

    Ziyu Li and Donghwan Shin. 2024. Mutation-based consistency testing for eval- uating the code understanding capability of llms. InProceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 150– 159

    2024

  15. [15]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  16. [16]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. InThirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=1qvx610Cu7

    2023

  17. [17]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems 36 (2024)

    2024

  18. [18]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  19. [19]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al . 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664(2021)

    Pith/arXiv arXiv 2021

  20. [20]

    Wei Ma, Shangqing Liu, Zhihao Lin, Wenhan Wang, Qiang Hu, Ye Liu, Cen Zhang, Liming Nie, Li Li, and Yang Liu. 2023. Lms: Understanding code syntax and semantics for code analysis.arXiv preprint arXiv:2305.12138(2023)

    Pith/arXiv arXiv 2023

  21. [21]

    Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating code summarization techniques: A new metric and an empirical characterization. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  22. [22]

    Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gallé. 2024. On leakage of code generation evaluation datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024. 13215– 13223

    2024

  23. [23]

    Roberto Minelli, Andrea Mocci, and Michele Lanza. 2015. I know what you did last summer-an investigation of how developers spend their time. In2015 IEEE 23rd international conference on program comprehension. IEEE, 25–35

    2015

  24. [24]

    Lefteris Moussiades and George Zografos. 2023. OpenAi’s GPT4 as coding assistant.arXiv preprint arXiv:2309.12732(2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models.arXiv preprint arXiv:2308.07124(2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2023. In-ide generation-based information support with a large language model.arXiv preprint arXiv:2307.08177(2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

    2002

  28. [28]

    Zeeshan Rasheed, Malik Abdul Sami, Muhammad Waseem, Kai-Kristian Kemell, Xiaofeng Wang, Anh Nguyen, Kari Systä, and Pekka Abrahamsson. 2024. AI- powered Code Review with LLMs: Early Results.arXiv preprint arXiv:2404.18496 (2024)

    arXiv 2024

  29. [29]

    Ehud Reiter. 2018. A structured review of the validity of BLEU.Computational Linguistics44, 3 (2018), 393–401

    2018

  30. [30]

    Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105–1116

    2021

  31. [31]

    Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. InProceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 27–43

    2022

  32. [32]

    David A Sousa. 2016. Primacy/recency effect

    2016

  33. [33]

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2024. Source code summarization in the era of large language models.arXiv preprint arXiv:2407.07959(2024)

    Pith/arXiv arXiv 2024

  34. [34]

    Weisong Sun, Yiran Zhang, Jie Zhu, Zhihui Wang, Chunrong Fang, Yonglong Zhang, Yebo Feng, Jiangping Huang, Xingya Wang, Zhi Jin, et al. 2025. Com- menting Higher-level Code Unit: Full Code, Reduced Code, or Hierarchical Code Summarization.arXiv preprint arXiv:2503.10737(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Balázs Szalontai, Gergő Szalay, Tamás Márton, Anna Sike, Balázs Pintér, and Tibor Gregorics. 2024. Large Language Models for Code Summarization.arXiv preprint arXiv:2405.19032(2024)

    Pith/arXiv arXiv 2024

  36. [36]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  37. [37]

    Bo Wang, Mingda Chen, Youfang Lin, Mike Papadakis, and Jie M Zhang. 2024. An exploratory study on using large language models for mutation testing.arXiv preprint arXiv:2406.09843(2024)

    arXiv 2024

  38. [38]

    Yang Wu, Yao Wan, Zhaoyang Chu, Wenting Zhao, Ye Liu, Hongyu Zhang, Xuanhua Shi, and Philip S Yu. 2024. Can Large Language Models Serve as Evaluators for Code Summarization?arXiv preprint arXiv:2412.01333(2024)

    Pith/arXiv arXiv 2024

  39. [39]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675(2019)

    Pith/arXiv arXiv 2019