{"id":"6912bd52-2963-4d16-bd33-9559d5d64051","arxiv_id":"2605.22526","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Empirical study finds coding agents produce fewer and less intense tangled refactorings than humans on Multi-SWE-bench; a refactoring-aware refinement improves compilability from 19.34% to 38.33% and resolves 2.79% more issues.","lead":"Coding agents for resolving software issues introduce tangled refactorings less often and with lower intensity than human developers, though with greater variety in types. A proposed refinement method that evaluates and selectively removes or repairs these refactorings raises compilability rates and resolves additional issues.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Refinement assessment of necessity/safety lacks grounding in original issue context or tests","rationale":"The reader's weakest assumption directly identifies the same point; the empirical findings on frequency and logistic regression are secondary to whether the downstream refinement step is sound. Full-text methods section would be needed to confirm absence of context retrieval, but the abstract alone already flags this as the load-bearing step.","tokens_in":1729,"tokens_out":309,"duration_ms":38606,"concrete_test":"Sample 30 patches where the refinement removed at least one refactoring; for each, re-insert the removed refactoring(s) into the refined patch, re-compile, and re-run the issue's test suite. If >15% of re-inserted cases still compile and pass tests, the necessity assessment is over-aggressive and the headline gains are not reliably attributable to the proposed method.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on the refinement approach correctly identifying and removing/repairing only unnecessary or unsafe tangled refactorings. The description indicates this assessment uses LLM-based judgment on the patch, but provides no explicit mechanism (e.g., prompts, rules, or retrieval) for incorporating the original issue description, failing tests, or commit intent. Without this, removals risk either leaving compilability issues or discarding refactorings that were required for the functional fix, undermining both the 19.34%→38.33% compilability gain and the 2.79% additional resolutions.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper conducts an empirical study on 3,691 valid patches generated by three coding agent frameworks using 12 LLMs on the Multi-SWE-bench dataset. It reports that agents introduce tangled refactorings less frequently (21.43% vs. 36.72%) and with lower intensity (0.66 vs. 1.75) than human developers, though with broader diversity of refactoring types. Logistic regression shows tangled refactorings are strongly associated with reduced compilability but not with functional correctness. The authors propose an LLM-based refactoring-aware refinement approach to assess necessity and safety of tangled refactorings and selectively remove or repair them, claiming this raises compilability from 19.34% to 38.33% while resolving an additional 2.79% of previously unresolved issues.","tokens_in":1832,"tokens_out":544,"duration_ms":37833,"significance":"If the refinement method can be shown to correctly distinguish necessary from unnecessary refactorings without discarding functional fixes or introducing new errors, the work supplies a concrete, deployable technique for improving the reliability of agent-generated patches in real issue-resolution workflows. The comparative statistics on agent versus human refactoring practices add to the growing literature on how LLM training corpora shape code-editing behavior. The study is a useful first step in characterizing side-effects of agentic code generation.","major_comments":[{"comment":"Description of the refactoring-aware refinement approach: the necessity and safety assessment is described as LLM-based judgment performed on the patch. No explicit mechanism (prompt template, retrieval step, or rule set) is provided for incorporating the original issue description, failing test cases, or resolution intent. Because the reported compilability gain (19.34% → 38.33%) and the additional 2.79% resolutions rest directly on the correctness of these removals/repairs, the absence of grounding context is load-bearing and requires either a detailed prompt appendix or an ablation that measures functional regression after refinement.","section":"Refinement approach section"}],"minor_comments":[{"comment":"The abstract states logistic regression results but supplies no model specification, variable list, or goodness-of-fit statistics; these details should appear in the main text or an appendix to support reproducibility.","section":"Abstract and §3"},{"comment":"Table or figure reporting the 3,691-patch breakdown by framework and LLM is missing; adding it would allow readers to judge whether the frequency and intensity differences are driven by particular model families.","section":"Empirical study section"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We address the major comment below and will incorporate the suggested clarifications and additional analysis in the revised manuscript.","responses":[{"response":"We agree that the current description of the refactoring-aware refinement approach is insufficiently detailed regarding the incorporation of grounding context. In the revised manuscript we will add the complete prompt template to an appendix. The template explicitly includes the original issue description, the failing test cases, and the resolution intent as part of the input provided to the LLM for necessity and safety assessment. We will also add an ablation study that reports functional correctness (pass rates on the original test suite) before and after refinement to confirm that the observed gains in compilability do not introduce regressions or discard functional fixes.","revision_made":"yes","referee_comment":"[Refinement approach section] Description of the refactoring-aware refinement approach: the necessity and safety assessment is described as LLM-based judgment performed on the patch. No explicit mechanism (prompt template, retrieval step, or rule set) is provided for incorporating the original issue description, failing test cases, or resolution intent. Because the reported compilability gain (19.34% → 38.33%) and the additional 2.79% resolutions rest directly on the correctness of these removals/repairs, the absence of grounding context is load-bearing and requires either a detailed prompt appendix or an ablation that measures functional regression after refinement."}],"tokens_in":1475,"tokens_out":315,"duration_ms":32594,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that coding agents mix in fewer and less intense tangled refactorings than human developers on the same tasks, yet those tangles still hurt compilability, and a post-processing step that drops or repairs some of them produces a clear lift in compilability plus a small gain in resolved issues.","headline":"Agents do fewer tangled refactorings than humans and a refinement pass lifts compilability from 19% to 38%, but the safety judgment looks thin on context.","tokens_in":2307,"tokens_out":138,"would_cite":false,"duration_ms":27865,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean","rs_theorem":"reality_from_one_distinction","paper_passage":"We propose RefUntangle, a refactoring-aware refinement approach that assesses the necessity and safety of tangled refactorings and selectively removes or repairs problematic operations."},{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"Logistic regression analysis further shows that tangled refactorings are strongly associated with reduced compilability"}],"headline":"Empirical SE study on agent refactorings has no overlap with RS forcing chain","alignment":"orthogonal","rationale":"Paper analyzes tangled refactorings in coding-agent patches on Multi-SWE-bench, logistic regression on compilability/correctness, and proposes LLM-based RefUntangle refinement. Central machinery (RefactoringMiner detection, necessity/safety assessment, patch diff editing) lies entirely in software maintenance and LLM post-processing. No J-cost, ratio symmetry, phi-ladder, 8-tick periodicity, or parameter-free constant derivation appears. Domain is orthogonal to the RS foundation theorems.","tokens_in":57385,"confidence":"high","tokens_out":294,"duration_ms":12728,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Coding agents introduce tangled refactorings less often than humans, but those they add reduce patch compilability; a targeted refinement step more than doubles the compilability rate.","keywords":["tangled refactorings","coding agents","issue resolution","compilability","refactoring","LLM agents","software patches","empirical study"],"falsifier":"Running the refinement on a new collection of agent patches and observing no rise in compilability or the appearance of new compilation errors.","tokens_in":2635,"feed_emoji":"🛠","tokens_out":612,"duration_ms":53355,"temperature":0.7,"pith_summary":"The paper analyzes thousands of patches generated by coding agents on real software issues and compares them to human patches. Agents produce tangled refactorings at lower frequency and intensity than humans, yet these refactorings still correlate strongly with failures to compile. The authors introduce a refinement method that checks each tangled refactoring for necessity and safety, then removes or repairs the problematic ones. Applying this method raises compilability from 19.34 percent to 38.33 percent and resolves a small additional share of previously unsolved issues.","feed_headline":"Refinement more than doubles compilability of agent patches","feed_subtitle":"Agents use fewer tangled refactorings than humans yet still suffer from them; selective removal lifts compilability from 19% to 38%.","key_machinery":"The refactoring-aware refinement approach, which identifies tangled refactorings in generated patches and judges their necessity and safety before deciding whether to remove or repair them.","core_discovery":"Tangled refactorings appear less frequently and with lower intensity in agent patches than in human ones, but they remain associated with sharply reduced compilability while showing no link to functional correctness. A refactoring-aware refinement approach that assesses necessity and safety of each tangled refactoring and then selectively removes or repairs the unsafe ones improves compilability from 19.34% to 38.33% and resolves 2.79% more issues.","pith_inferences":["Agents could be prompted or trained to limit refactoring to only what is required for the fix.","The same necessity-and-safety check might help in other code-change tasks such as feature implementation.","Training data drawn from open-source repositories may explain why agents still produce some tangled refactorings."],"forward_implications":["Tangled refactorings reduce compilability of agent-generated patches.","Agents display a wider range of refactoring types than human developers.","The refinement step can be applied after any agent run to raise the share of compilable patches.","Logistic regression shows the negative effect on compilability is independent of functional correctness."],"fun_headline_variants":["Coding agents introduce fewer tangled refactorings than humans","Tangled refactorings lower compilability in agent patches","Refactoring aware approach improves compilability from 19% to 38%","Selective refactoring fixes resolve additional issues in agents"],"cache_read_input_tokens":64,"weakest_assumption_plain":"That tangled refactorings found in agent patches can be judged for necessity and safety by the refinement method without missing important context or creating fresh problems.","fun_headline_variants_meta":{"raw":{"variants":["Coding agents introduce fewer tangled refactorings than humans","Tangled refactorings lower compilability in agent patches","Refactoring aware approach improves compilability from 19% to 38%","Selective refactoring fixes resolve additional issues in agents"]},"model":"grok-4.3","cost_usd":0.011915,"raw_usage":{"total_tokens":5146,"prompt_tokens":706,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":119153000,"prompt_tokens_details":{"text_tokens":706,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4378,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":706,"tokens_out":62,"duration_ms":41764,"temperature":1.0,"reasoning_tokens":4378,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T03:54:00.845261+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the refinement on a new collection of agent patches and observing no rise in compilability or the appearance of new compilation errors.","supporting_citations":[],"review_version":1}