{"id":"344019dc-dcde-4e70-984f-ccd03deb002d","arxiv_id":"2501.09879","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RETESTER generates refactoring-preserving Java variants from historical bug reports using an LLM and finds 18 new bugs in Eclipse and IntelliJ refactoring engines.","lead":"A new tool called RETESTER uses a large language model to generate mutated Java programs from historical bug reports and tests them on Eclipse and IntelliJ refactoring engines. It found 18 new engine bugs, seven of which were confirmed by developers and three fixed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing risk is not the choice of three characteristics but whether the 18 oracle violations are genuine refactoring-engine bugs: most lack developer confirmation and Table VIII's statuses do not match the 'seven confirmed, three fixed' claim.","rationale":"The reader's weakest assumption focuses on whether the three selected characteristics and five seeds are representative. That is a legitimate generality concern, but it is not the most load-bearing for the paper's headline. Even if the characteristics are not representative, the discovered bugs could still be real; the central claim would survive. Conversely, if many of the 18 reported bugs are not genuine refactoring-engine defects, the approach's headline result collapses regardless of characteristic choice. The paper does provide external evidence: 7 developer responses in the issue trackers, and the three fixed issues are independent confirmation. But Table VIII's status distribution (4 confirmed, 3 fixed, 11 submitted) does not cleanly match the abstract's 'seven confirmed and three fixed,' and 11 bugs remain unconfirmed. Given that Definition 1's RPT notion is weak (it preserves only applicability, not semantics), and the primary oracle is the Uncompilable Oracle, the risk of false positives from LLM-mutated programs is non-trivial. My recommendation is to keep the reader's CONDITIONAL verdict: the approach is plausible and the external confirmations are encouraging, but the paper should release the artifacts, reconcile the status counts, and ideally provide an independent reproduction before the 18-bug claim is taken at face value.","tokens_in":16658,"tokens_out":7501,"duration_ms":80743,"concrete_test":"Re-run the evaluation independently from the promised artifact: use the exact prompts in Tables I, III, and V with gpt-4o-mini to regenerate the 150 variants, compile with JDK 22.0.1, apply the five refactorings in Eclipse 2024-09 and IntelliJ IDEA 2024.2.4, and apply the three oracles from Section III-F without author triage. Compare the per-row bug counts with Table VII and the statuses in Table VIII; if fewer than 18 unique bugs reproduce, or if the confirmed/fixed counts differ from seven/three, the central claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the discovery of 18 new refactoring-engine bugs. That claim is only as strong as the validation of each oracle violation, and most of that validation is internal to the paper. Table VII shows 20 Uncompilable, 1 Warning-Status, and 2 Differential violations, but the 'bugs' column reports 18 unique bugs after manual triage; Table VIII lists 4 'Confirmed', 3 'Fixed', and 11 'Submitted' issues, while the abstract says 'seven of them were confirmed ... and three were fixed.' The statuses are at least ambiguous and possibly inconsistent. More importantly, Definition 1 (Section III-E) defines refactoring-preserving only as the refactoring operation remaining applicable to some element in the variant; it does not require the variant to be semantically equivalent to the seed or the refactoring to be valid. The Uncompilable Oracle (Section III-F) then treats any syntax error in the refactored program as a bug. For 11 of the 18 bugs the only check is the authors' manual classification, since those issues were not confirmed or fixed by developers. This is a real soft spot: an LLM-generated variant that changes program semantics could make a correct engine legitimately refuse or produce a different result, and the paper does not provide an automated equivalence or precondition check to rule this out. The small seed set and the three hand-picked characteristics are secondary; the decisive question is whether the reported bug list is real.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RETESTER, an LLM-based approach to testing Java refactoring engines. It mines historical bug reports from Eclipse and IntelliJ IDEA, extracts seed programs and refactoring information using few-shot prompting, and then uses chain-of-thought prompts to generate variants that incorporate error-prone language characteristics (lambda, generics, anonymous classes) while preserving applicability of the original refactoring. The variants are applied to the latest engine versions and checked with uncompilable, warning-status, and differential oracles. The authors report discovering 18 new bugs (15 in Eclipse, 3 in IntelliJ IDEA), with 7 confirmed and 3 fixed by developers at submission time. An ablation shows that using an abstract program template yields more bugs than mutating the concrete input program.","tokens_in":16931,"tokens_out":8130,"duration_ms":78485,"significance":"If the reported bugs are genuine, the paper would be a valuable demonstration that LLMs can translate historical bug-triggering structures into new test programs for increasingly complex IDE refactoring engines, complementing earlier template-based and random-testing approaches. The dataset of 167 compilable Java programs extracted from bug reports, the 98/100 extraction accuracy, and the reproducible workflow are concrete strengths. However, the central claim rests on the manual validation of oracle violations, and the paper's current presentation does not adequately rule out false positives caused by semantic changes in LLM-generated variants, nor does it reconcile the reported bug statuses. These issues must be resolved before the results can be taken at face value.","major_comments":[{"comment":"The refactoring-preserving transformation only requires that a program element exists in the variant on which the refactoring operation remains applicable; it does not require the variant to be semantically equivalent to the seed or the refactoring to be valid. The Uncompilable and Differential oracles then treat any syntax error or output difference as a potential engine bug. Because the LLM variants may change program semantics, an engine's refusal or different result could be correct for the modified program. The paper must either strengthen Definition 1 to require behavioral preservation (or at least a recorded proof that the refactoring precondition holds and the transformation is semantics-preserving) or provide an automated or manual semantic-equivalence check for each reported issue. In particular, for the 11 issues that developers have not confirmed or fixed, the evidence currently rests entirely on the authors' manual classification.","section":"Section III-E, Definition 1; Section III-F"},{"comment":"The reported confirmation counts are inconsistent. The abstract and conclusion say 'seven of them were confirmed ... and three were fixed,' but Table VIII lists only four issues with status 'Confirmed' and three with 'Fixed'; the text in RQ1 says both 'seven have been confirmed' and 'four out of the eighteen bugs have been officially confirmed.' If 'confirmed' is meant to include 'fixed,' the table should use a single status field; if not, the abstract number is wrong. The authors should correct this and give the exact developer response for each issue.","section":"Abstract; Section V-A; Table VIII"},{"comment":"The baseline comparison does not faithfully represent Gligoric et al.'s approach. The baseline is implemented by having an LLM propose refactoring operations on the seed input programs, rather than applying refactorings at randomly chosen program elements in real projects as in the original work. With only 50 variants per configuration, the finding that the baseline reports zero bugs is best interpreted as a no-mutation ablation, not as evidence that RETESTER outperforms the state of the art. The claim 'Without performing mutation on the input program, Gligoric's approach fails to find any bugs' is therefore overstated.","section":"RQ2, Table IX"},{"comment":"The Abstract and Contributions state that the data are open-sourced, but the Data Availability section says reference [25] is 'To be open-sourced.' This makes the dataset and the 18 bug artifacts unverifiable. Provide an accessible link or state a clear availability policy.","section":"Data Availability; reference [25]"}],"minor_comments":[{"comment":"The last column header '18 (15)' is unexplained in the footnote; add a note that the parenthetical is the number of unique Eclipse bugs after removing three overlaps between the template and no-template conditions.","section":"Table VII"},{"comment":"Section IV-A states that 150 variants are generated, while Table VII totals 300 variants; clarify that the additional 150 come from the RQ4 ablation (Template=N rows).","section":"Section IV-A; Table VII"},{"comment":"The phrase 'lst two columns' should read 'last two columns.'","section":"Section V-D"},{"comment":"The term 'refactoring-preserving' is misleading because Definition 1 only preserves applicability, not behavior; consider renaming it to 'refactoring-applicability-preserving' or explicitly discussing the distinction.","section":"Section III-E"},{"comment":"The three code blocks in Figure 3 are difficult to read; increase the font size and separate the seed, template, and variant visually.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real problem and the methodology is plausible, but the validation of reported bugs and the inconsistent confirmation counts are the main hurdles. I recommend major revision rather than rejection because the issues are fixable with additional analysis and disclosure. If the authors can provide a reproducible artifact and verify each of the 18 bugs (or adjust the confirmed count accordingly), the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuine first: applying LLMs to generate refactoring-preserving variants from historical bug reports, and the template ablation is clean. Second, the headline number, 18 new bugs, is not as solid as it looks: the Uncompilable Oracle counts any post-refactoring syntax error as a bug, and Definition 1 only requires the refactoring to remain applicable, not that the variant preserve semantics. For the 11 issues without developer confirmation, we are taking the authors' word that the engine is wrong.\n\nWhat is actually new: RETESTER is the first LLM-based history-driven testing approach for refactoring engines. The pipeline, from mining bug reports, extracting compilable seeds with LLM (98/100 accuracy on a sample of 100), to mutation prompts with error-prone characteristics and a structure template, is sensible. The ablation showing template use roughly doubles the bug yield is informative. The two-annotator labeling with Kappa 0.93 is carefully done. The dataset of 167 compilable programs is a useful artifact, though the data availability reference is just a placeholder.\n\nThe soft spots are real. The semantic-equivalence gap is the main one. If an LLM-changed variant changes behavior, a correct engine may legitimately refuse or produce a different program; the paper does not check equivalence or preconditions, so the Uncompilable and Differential oracles over-report. The internal count inconsistency is concrete: the abstract says seven confirmed and three fixed, but Table VIII lists four confirmed and three fixed. The evaluation scale is small (five seeds, five refactoring types) and only three error-prone characteristics from the authors' own prior study are used—self-citation is not a problem, but independent grounding is missing. No code or data are actually released yet.\n\nWho this is for: people working on refactoring-engine testing, compiler testing, and LLM-generated test inputs. They will get a clear technique description and a cautionary case study.\n\nMy recommendation: send it to peer review, but with a referee who presses on the oracle definition and asks for an equivalence check or a downgraded claim for unconfirmed bugs. Fix the count inconsistency and release the artifacts, and this becomes a useful paper.","headline":"Useful first step in LLM-based refactoring-engine testing, but the bug claims rest on a weak equivalence oracle and an internal count inconsistency.","tokens_in":17497,"tokens_out":4430,"would_cite":true,"duration_ms":38237,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RETESTER shows that LLM-generated, history-seeded refactoring-preserving variants can uncover 18 new bugs in the latest Eclipse and IntelliJ IDEA refactoring engines.","keywords":["refactoring engine testing","LLM-based test generation","differential testing","historical bug reports","input program mutation","refactoring-preserving transformation","Eclipse JDT","IntelliJ IDEA"],"falsifier":"Run the same five-seed, three-characteristic pipeline on a held-out set of historical bug reports whose refactoring types are not among the five seeds, such as Rename, Move, or Change Signature, and count newly confirmed developer issues; if the bug yield does not exceed the no-mutation baseline, the claim that historical-bug-report-driven LLM mutation generalizes across refactoring types would be refuted.","tokens_in":16425,"feed_emoji":"🐛","tokens_out":7221,"duration_ms":64073,"temperature":0.7,"pith_summary":"This paper proposes RETESTER, a way to test refactoring engines by having a large language model generate new bug-triggering Java programs from historical bug reports. The key move is to preserve the applicability of the original refactoring while mutating the program with an error-prone construct such as lambda expressions, Java generics, or anonymous classes, so that any difference in the engine's output indicates a bug. On five seed reports covering five refactoring types, RETESTER generated 150 variants and found 18 previously unreported bugs in the latest Eclipse and IntelliJ IDEA refactoring engines; seven were confirmed by developers and three fixed by submission time. The paper also shows that abstracting the seed program into a structural template, rather than mutating the concrete code, is what lets the LLM explore diverse edge cases.","feed_headline":"LLM-crafted variants expose 18 new IDE refactoring bugs","feed_subtitle":"Bug reports seed variants that exposed real compile errors in Eclipse and IntelliJ IDEA.","key_machinery":"The central mechanism is the refactoring-preserving transformation (RPT): a mutation Trans(P1)=P2 is valid only if the same refactoring operation O that applied to program element E1 in the seed still applies to some element E2 in the variant. This condition keeps generated programs inside the refactoring engine's precondition space so that engine misbehavior can be attributed to the refactoring itself. RETESTER operationalizes RPT with chain-of-thought prompts: the LLM first reads a refactoring definition, then an input-program structure template extracted from a historical bug report, then one error-prone characteristic, and must output the variant plus the elements and procedures for refactoring. Variants are filtered by javac compilation and by checking refactoring applicability, then the same refactoring is run in both IDEs under three oracles: uncompilable output, differing warning status, and differing refactored programs.","core_discovery":"The paper's central claim is that a history-driven, LLM-based mutation pipeline can find real bugs in current refactoring engines without manually designing templates for each refactoring type. Starting from fixed historical bug reports, RETESTER extracts the input program, the refactoring type and procedure, and an abstract input-program structure template. It then prompts the LLM, in a chain-of-thought style, to produce a refactoring-preserving variant that introduces one of three error-prone Java features. Applying the same refactoring to each variant in Eclipse JDT (2024-09) and IntelliJ IDEA (2024.2.4) and using differential oracles, the authors found 18 new bugs, 15 unique in Eclipse and 3 in IntelliJ IDEA; seven were confirmed by developers and three were fixed. An ablation without the template found only six bugs, with three overlapping, and a no-mutation baseline found none, supporting the claim that the combination of historical structure and targeted mutation is what exposes the bugs.","pith_inferences":["Beyond the paper: combining the three error-prone characteristics in one variant, such as an anonymous class inside a generic type, could reveal interaction bugs that single-characteristic mutations miss; the paper leaves such combinations for future work.","Beyond the paper: the same template-extraction and refactoring-preserving prompt recipe could be applied to other languages or to compiler and JIT testing, where history-driven seeds are already known to help.","Beyond the paper: refactoring each variant by hand in the IDE limits throughput; automating the refactoring step would let the approach scale to thousands of variants and could change the measured bug yield.","Beyond the paper: the 15-to-3 split between Eclipse and IntelliJ IDEA may reflect differences in precondition checking rather than an intrinsic quality ranking, so the finding should not be read as a verdict on which IDE is better."],"forward_implications":["If RETESTER's results hold, refactoring engines in current IDEs still emit uncompilable code on small programs that combine nested classes with lambda expressions, Java generics, or anonymous classes.","The extracted input-program structure templates can be reused by template-based testing tools as historically grounded templates instead of hand-crafted ones.","The pipeline extends to additional refactoring types simply by adding more historical bug reports, without redesigning the mutation prompts for each type.","Bug-triggering seeds tend to be small (median 9 lines of code), so test generators should favor small, structurally complex programs over larger, simpler ones.","Pull Up and Inline Method produced the most bugs in this sample, giving engine developers concrete areas to prioritize."],"supporting_citations":[{"why":"Supplies the empirical basis for choosing lambda expressions, Java generics, and anonymous classes as the most error-prone input program characteristics, and for compile errors being the dominant symptom.","marker":"[10]"},{"why":"Provides the earlier template-based testing approach and the uncompilable, warning-status, and differential oracles that RETESTER adopts for judging refactoring engine bugs.","marker":"[11]"},{"why":"Gives the systematic real-project refactoring testing baseline that RETESTER compares against; applying it to the same seeds found zero bugs.","marker":"[18]"},{"why":"Is the Eclipse pull-up bug report used as the motivating example and one of the seed reports, showing failure of inner-class method resolution.","marker":"[27]"},{"why":"Is one of the newly submitted Eclipse bug reports produced by the anonymous-class variant, evidence that the pipeline triggers real engine bugs.","marker":"[29]"}],"fun_headline_variants":["Historical bugs + LLM mutations expose 18 new refactoring bugs","LLM turns historical bug reports into 18 new refactoring bugs","LLM-generated tests from past bugs find 18 new refactoring bugs","18 new refactoring bugs traced to LLM mutations from bug reports"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bug counts depend on the premise that lambda expressions, Java generics, and anonymous classes, chosen from an earlier empirical study and applied to only five seed reports, are the dominant error-prone traits that trigger refactoring-engine bugs; if the real triggers lie elsewhere, the approach would find far fewer bugs on other inputs.","fun_headline_variants_meta":{"raw":{"variants":["Historical bugs + LLM mutations expose 18 new refactoring bugs","LLM turns historical bug reports into 18 new refactoring bugs","LLM-generated tests from past bugs find 18 new refactoring bugs","18 new refactoring bugs traced to LLM mutations from bug reports"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":4045,"prompt_tokens":967,"completion_tokens":3078,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":2998}},"tokens_in":583,"tokens_out":3078,"duration_ms":19932,"temperature":1.0,"reasoning_tokens":2998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:34:48.101930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-seed, three-characteristic pipeline on a held-out set of historical bug reports whose refactoring types are not among the five seeds, such as Rename, Move, or Change Signature, and count newly confirmed developer issues; if the bug yield does not exceed the no-mutation baseline, the claim that historical-bug-report-driven LLM mutation generalizes across refactoring types would be refuted.","supporting_citations":[{"cited_title":"An Empirical Study of Refactoring Engine Bugs","cited_arxiv_id":"2409.14610","evidence_quote":"Supplies the empirical basis for choosing lambda expressions, Java generics, and anonymous classes as the most error-prone input program characteristics, and for compile errors being the dominant symptom."},{"cited_title":"Automated testing of refactoring engines,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier template-based testing approach and the uncompilable, warning-status, and differential oracles that RETESTER adopts for judging refactoring engine bugs."},{"cited_title":"Systematic testing of refactoring engines on real software projects,","cited_arxiv_id":null,"evidence_quote":"Gives the systematic real-project refactoring testing baseline that RETESTER compares against; applying it to the same seeds found zero bugs."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Is the Eclipse pull-up bug report used as the motivating example and one of the seed reports, showing failure of inner-class method resolution."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Is one of the newly submitted Eclipse bug reports produced by the anonymous-class variant, evidence that the pipeline triggers real engine bugs."}],"review_version":1}