{"id":"9e23ca48-a0e0-4961-9228-1a457db6c74a","arxiv_id":"2602.17838","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Mutation-based testing shows LLM-generated code summaries frequently fail to reflect behavior-changing edits, with accuracy dropping as code complexity rises.","lead":"A new evaluation method for LLM code summaries injects small code mutations and checks whether the model's explanation changes to reflect the new behavior. In tests across 624 mutated samples, GPT-4 often kept describing the original intent, though newer GPT-5.2 caught most mutations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified behavior-change assumption: equivalent mutants would be mislabeled as summarization failures.","rationale":"The paper's central claim is that mutation analysis systematically tests whether summaries track program behavior. That claim requires that every mutation actually changes behavior; otherwise, an unchanged summary is correct, not a failure. The paper reports no dynamic verification of behavior change, despite this being standard practice in mutation testing (where equivalent mutants are explicitly identified and removed). The single-annotator and human-judgment issues identified by the reader are real but secondary: they concern the measurement of the dependent variable, whereas the equivalent-mutant issue concerns the ground truth itself. If the ground truth is contaminated, all downstream percentages are suspect. The proposed test is feasible because the programs are small and self-contained; running them on a battery of inputs is straightforward. If the test shows that all or nearly all mutants are non-equivalent, the concern is resolved and the reader's emphasis on annotator reliability becomes the primary remaining threat. If a substantial fraction are equivalent, the paper's empirical conclusions require recalibration. Thus the verdict remains CONDITIONAL, pending this verification.","tokens_in":13776,"tokens_out":5771,"duration_ms":56907,"concrete_test":"For each of the 324 synthetic mutants and 150 LBPP mutants, execute the original and mutated programs on a comprehensive input suite (e.g., exhaustive inputs up to a small size, plus random tests). If any mutant produces identical outputs on all tests for its original, classify it as equivalent and recompute all reported percentages after excluding equivalent mutants. If the recomputed percentages differ materially (e.g., >10 percentage points), the central empirical claims need revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The methodology's dependent variable—whether a mutation can be detected from the paired summaries—presupposes that each injected mutation actually changes the program's observable behavior. If a mutant is equivalent (i.e., produces identical outputs for all inputs), then the correct summary should not change, yet the paper classifies an unchanged summary as a 'negative' failure. The paper never executes the mutated programs to verify behavior changes; Section 6.2 asserts mutations 'change the behavior of the code in a meaningful way' but supplies no test or filter. This threatens the denominator of every reported percentage (76.5%, 49.3%, 85.3%, and all per-type rates). If even a modest fraction of the 324+150 mutants are equivalent, the accuracy figures are systematically deflated, and the central claim that LLMs fail to track behavior is overstated. This is logically prior to the annotator-reliability issue: perfect annotation cannot correct mislabeled ground truth.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a mutation-based methodology for evaluating LLM-generated code summaries. For each code sample, the authors generate an LLM summary, inject a targeted mutation, generate a summary for the mutated code, and then ask human annotators whether the mutation can be identified by comparing the two summaries. A positive classification means the mutation is detectable from the summaries; a negative means it is not. The methodology is applied in three experiments: (1) 324 mutations across 12 synthetic programs spanning four complexity levels, reporting a decline in detection from 76.5% (single function) to 17.3% (multi-threaded); (2) 150 mutations across 50 human-written LBPP programs, reporting 49.3% detection for GPT-4; and (3) the same 150 LBPP mutations evaluated with GPT-5.2, reporting 85.3% detection. The paper concludes that LLM summaries often track intended algorithmic patterns rather than actual mutated behavior, and argues that mutation analysis is a practical framework for comparing models and configurations.","tokens_in":13980,"tokens_out":5870,"duration_ms":63088,"significance":"The proposed methodology is a useful departure from n-gram-based summary metrics: it creates a behavioral probe of whether summaries track code changes, and the RQ1 experiment includes dual annotation with high inter-rater agreement (96.6%, Cohen's kappa = 0.928). The qualitative failure-mode analysis (missing details vs. hallucinated intent) is instructive, and the replication package is a tangible asset. If the validity concerns are resolved, the approach could provide a repeatable evaluation tool for code summarization. However, the three main quantitative claims suffer from load-bearing validity problems: the mutated programs are never executed to confirm that the mutations change behavior, the dependent variable is human detectability of a mutation from paired summaries rather than summary correctness, and the RQ2/RQ3 headline numbers rest on a single unblinded annotator. These issues are substantial but addressable, so the contribution is promising rather than invalid.","major_comments":[{"comment":"The dependent variable presupposes that every injected mutation changes observable behavior. Section 6.2 states that 'what matters for our analysis is that the mutation changes the behavior of the code in a meaningful way,' but no execution, test, or formal equivalence check is reported. If any mutant is equivalent (same I/O behavior for all inputs), the correct summary should not change, yet the paper counts an unchanged summary as a negative. This threatens the denominator of every reported rate (76.5%, 49.3%, 85.3%, Table 1). Please run the original and mutated programs on a test suite or otherwise verify behavioral change, exclude or quantify equivalent mutants, and re-analyze the data.","section":"Section 4, Section 6.2"},{"comment":"The paper equates 'mutation detectable from paired summaries' with 'summary accuracy.' A summary can be accurate but too coarse to mention a mutated detail; the paper's own first failure mode, 'summary is too abstract it omits the mutated logic entirely,' is treated as a failure without arguing that a correct summary must encode every behavioral change at that granularity. Conversely, two summaries can differ for reasons unrelated to mutation. The headline percentage is therefore better described as mutation sensitivity, not summary correctness. Please either reframe the claims as sensitivity to behavioral perturbations or add an independent human assessment of whether the mutated summary is actually incorrect.","section":"Section 4, Section 5.1, Abstract"},{"comment":"RQ2 and RQ3 classifications were performed by a single author, using the same criteria as RQ1 but on new data and on a second model's outputs. The RQ1 kappa of 0.928 does not establish reliability for the LBPP summaries or for GPT-5.2, and the annotator was not reported to be blinded to which model produced each summary. The 49.3% and 85.3% results are central to the paper. Please add a second annotator on a random subset (or all of RQ2/RQ3), report agreement, and blind the model identity during annotation.","section":"Section 6.1, Section 5.2, Section 5.3"},{"comment":"The chi-square test for complexity (chi-square(3)=69.04, p<0.001, V=0.462) treats 324 mutations as independent, but the mutations are nested within 12 programs (27 per program). Program-level correlation can inflate significance and understate uncertainty, and the same issue applies to the mutation-type and location tests and to the Mann-Whitney test on LOC. Please use a mixed-effects model, cluster-robust inference, or a program-level permutation test, or at least report intraclass correlation, to support the complexity and size conclusions.","section":"Section 5.1"},{"comment":"The definition of a negative outcome conflates two distinct cases: (a) the mutated summary is inaccurate because it describes original behavior, and (b) both summaries are accurate but too high-level to expose the mutation. The text says, 'if neither summaries explicitly described the mutated behaviour. That is, if the original and mutated summary both accurately describe the original and mutated code.' The second clause does not follow from the first: a summary that fails to describe the mutated behavior is not accurately describing the mutated code. Clarify the definition and separate these two cases in the analysis, since they have different implications.","section":"Section 4.2"}],"minor_comments":[{"comment":"The caption says 'The < sign on line 3 was changed to > on line 8,' but the code shows the original uses '>' and the mutant uses '<'. The direction is reversed.","section":"Listing 4 caption"},{"comment":"The caption says 'sample02/desc/b2 from sample01' and the text describes merge sort, but the code shown is a union-find 'find' function. The sample identifier and code snippet do not match.","section":"Listing 5 caption"},{"comment":"The caption says the sample is 'val_e_2 from sample05,' but the function shown is 'get_min' from a heap class; make the naming consistent with the sample identifiers used elsewhere.","section":"Section 1, Listing 1 caption"},{"comment":"The sentence 'the point here is not to model typical bugs, it is to make sure the code's behavior actually changes' is an assertion, not a verification. It would be stronger to cite the execution results requested in the major comment.","section":"Section 6.2"},{"comment":"The phrase 'LBPP was explicitly designed to be uncontaminated problems and solutions were verified not to overlap' is awkward; consider rewording and citing the specific verification procedure.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is candid about its limitations, but the limitations are not merely hedges: the unverified behavioral change of the mutants and the single-annotator RQ2/RQ3 labeling directly affect the headline percentages. I recommend major revision rather than rejection because the fixes are within scope: execute mutants and filter equivalents, reframe the metric as mutation sensitivity or validate it against summary correctness, and add a second annotator for the LBPP experiments. The proposed methodology itself is a reasonable contribution to code-summarization evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper is worth reading and worth refereeing, but the headline numbers need a caveat. The core idea is a real extension of prior mutation-based work: instead of asking whether an LLM can detect an inconsistency between code and a given description, the authors ask whether the model's own summary updates when the code is mutated. That's a sensible way to test whether summaries track behavior, and the three experiments give a useful comparison across program complexity, mutation type/location, and model generations.\n\nWhat works: the controlled synthetic dataset (12 programs, 324 mutations across SF/SC/MC/MT) is a real effort, the RQ1 human evaluation has high inter-rater reliability (κ=0.928), and the failure-mode analysis—summaries being too abstract vs. describing intended behavior—is genuinely informative. The replication package is good. The main empirical pattern (detection rate drops from 76.5% for single functions to 17.3% for multi-threaded code) matches the kinds of qualitative examples they show.\n\nThe soft spots: the stress-test concern is legitimate. The dependent variable is whether a human can spot the mutation from the paired summaries. If a mutant is equivalent—meaning it doesn't change the program's observable behavior—then the correct summary should not change, and the paper would count that as a failure. Section 6.2 asserts the mutations 'change the behavior of the code in a meaningful way' but there's no execution-based check or filter. Even a small share of equivalent mutants would deflate the reported percentages and overstate how often LLMs miss real behavioral changes. This isn't fatal, but it means the specific numbers should be treated as rough rather than measured until the mutants are validated.\n\nA related issue: the metric is 'can I detect the change from the summaries,' not 'is the summary correct.' A summary can be accurate yet too coarse to mention a mutated detail. Treating that as a failure is a defensible choice, but calling it 'accuracy' is a bit of a stretch and needs more defense.\n\nFinally, RQ2/RQ3 were classified by a single author with no reported inter-rater reliability. The paper's justification (high agreement in RQ1) is reasonable but not a substitute for at least a sample-based reliability check on the new data.\n\nBottom line: this is a solid, useful empirical study. It deserves peer review. The main revisions should be to verify the mutants change behavior (e.g., with tests or execution traces), report reliability for the later RQs, and reframe the metric as 'mutation-detection' or defend the accuracy framing more carefully.\n\nI'd bring it to a reading group as a good example of a mutation-based evaluation methodology. I'd cite it once the mutant-validation issue is addressed. Send it to review.","headline":"Useful mutation-based evaluation framework for LLM code summaries, but the headline accuracy rates depend on an unverified assumption that every mutant actually changes behavior.","tokens_in":14432,"tokens_out":3519,"would_cite":false,"duration_ms":31481,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that mutation-based testing—changing a program's behavior and checking whether the model's summary changes—can expose when large language models describe what code looks like it should do rather than what it actually does.","keywords":["mutation analysis","code summarization","large language models","program behavior vs intent","evaluation methodology","GPT-4","summary accuracy"],"falsifier":"Take the 150 human-written-program mutation pairs and have a fresh panel of independent annotators classify them with the same rubric without seeing the authors' labels. If their agreement with the published labels is low (for instance, Cohen's kappa below 0.6), the accuracy rates are not stable ground truth. A second falsifier: if expert human-written summaries of the same mutated programs fail the two-summary comparison as often as the model summaries, then the test is measuring summary detail, not correctness.","tokens_in":13695,"feed_emoji":"🧬","tokens_out":4206,"duration_ms":40590,"temperature":0.7,"pith_summary":"This paper tries to establish a way to test whether LLM-generated code summaries are actually grounded in program behavior. The method: summarize the code, make a small behavior-changing edit (a mutation), summarize the edited code, and ask whether a human can spot the edit by comparing the two summaries. Across 624 mutation-summary comparisons, the authors report that accuracy falls sharply as code grows more complex—from 76.5% for single functions to 17.3% for multithreaded systems—and that GPT-4 misses roughly half of mutations in human-written programs while GPT-5.2 misses about 15%. If correct, this gives developers and researchers a repeatable way to check whether a summary follows the logic of the code rather than surface patterns.","feed_headline":"Mutation test: LLM summaries miss most behavior changes","feed_subtitle":"Only 17.3% of summaries reflected a behavior-changing edit in multithreaded code; GPT-5.2 caught 85.3% on simpler programs.","key_machinery":"The load-bearing object is the mutation—a small, behavior-changing edit (statement, value, or decision) placed at a chosen location—used as a probe. The evaluation pipeline is: (1) generate mutants from original code; (2) generate a fresh, temperature-0 summary for the original and each mutant with the same prompt; (3) have human evaluators decide whether the change can be identified by comparing summaries. The mutation is doing the work of turning 'does the summary reflect the code' into a concrete yes/no question: if the summary is keyed to actual behavior, a behavior change must show up in the summary.","core_discovery":"The central claim is that mutation analysis provides a direct probe for whether a summary tracks behavior: if a change that alters what the program does is not reflected in the corresponding summary, that summary is unreliable regardless of how fluent it reads. The authors find a consistent failure mode in which models describe the intended or canonical algorithm—merge sort, heap operations, union-find—even after the mutation breaks that behavior. They operationalize this with a positive/negative classification: positive if the behavioral change can be identified from the two summaries, negative if not. The strongest empirical result is a steep complexity gradient, with accuracy falling mono","pith_inferences":["The method could be automated by using a second large language model as the comparator, replacing human judgment, which would make it scalable to entire repositories—though this would require validating that the model judge agrees with human labels.","The same protocol could become a contamination-resistant benchmark for code summarization by standardizing mutations across programming languages and measuring detection rates, an alternative to leaderboard-style benchmarks.","A testable extension: prompt the model to explicitly identify the mutation after summarizing; if the model can name the change but does not incorporate it into the summary, that would separate extraction failure from verbalization failure.","The finding that hard-coded string literals can anchor a model's interpretation suggests a specific intervention: perturb comments, names, and literals to see whether summaries track semantics or lexical cues."],"forward_implications":["If mutation-based evaluation is sound, it offers a repeatable stress test for any code summarizer, independent of reference summaries or n-gram overlap metrics.","The reported complexity gradient implies real-world, multi-component codebases are exactly where LLM summaries are least trustworthy.","The persistent failure mode—describing canonical intent over mutated behavior—implies that fluent summaries can actively mislead code review and documentation use.","The GPT-4 to GPT-5.2 comparison (49.3% to 85.3% on human-written single-function programs) suggests the method can measure generational progress under identical conditions.","Because the human-written programs are mostly single functions, the 85.3% figure should be read as a best-case; the authors expect performance to degrade on larger, interconnected code."],"fun_headline_variants":["LLM summaries miss most behavior changes in code mutations","Only 17% of LLM summaries reflect multithreaded code mutations","Mutation test reveals LLM summaries describe intent, not behavior","GPT-5.2 improves mutation detection but still confuses patterns with details","LLM code summaries fail to track behavior changes in complex systems"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a human's ability to detect the mutation by comparing two summaries is a faithful measure of summary accuracy—if a summary is accurate but too coarse to mention a mutated detail, the method counts it as a failure, and that conflation would change every reported rate.","fun_headline_variants_meta":{"raw":{"variants":["LLM summaries miss most behavior changes in code mutations","Only 17% of LLM summaries reflect multithreaded code mutations","Mutation test reveals LLM summaries describe intent, not behavior","GPT-5.2 improves mutation detection but still confuses patterns with details","LLM code summaries fail to track behavior changes in complex systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000365,"raw_usage":{"total_tokens":1850,"prompt_tokens":840,"completion_tokens":1010,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":584,"tokens_out":1010,"duration_ms":10365,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:05:32.176700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 150 human-written-program mutation pairs and have a fresh panel of independent annotators classify them with the same rubric without seeing the authors' labels. If their agreement with the published labels is low (for instance, Cohen's kappa below 0.6), the accuracy rates are not stable ground truth. A second falsifier: if expert human-written summaries of the same mutated programs fail the two-summary comparison as often as the model summaries, then the test is measuring summary detail, not correctness.","supporting_citations":[],"review_version":1}