REVIEW 10 cited by
An Empirical Study on the Code Refactoring Capability of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
An Empirical Study on the Code Refactoring Capability of Large Language Models
read the original abstract
Large Language Models (LLMs) have shown potential to enhance software development through automated code generation and refactoring, reducing development time and improving code quality. This study empirically evaluates StarCoder2, an LLM optimized for code generation, in refactoring code across 30 open-source Java projects. We compare StarCoder2's performance against human developers, focusing on (1) code quality improvements, (2) types and effectiveness of refactorings, and (3) enhancements through one-shot and chain-of-thought prompting. Our results indicate that StarCoder2 reduces code smells by 20.1% more than developers, excelling in systematic issues like Long Statement and Magic Number, while developers handle complex, context-dependent issues better. One-shot prompting increases the unit test pass rate by 6.15% and improves code smell reduction by 3.52%. Generating five refactorings per input further increases the pass rate by 28.8%, suggesting that combining one-shot prompting with multiple refactorings optimizes performance. These findings provide insights into StarCoder2's potential and best practices for integrating LLMs into software refactoring, supporting more efficient and effective code improvement in real-world applications.
Forward citations
Cited by 10 Pith papers
-
SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair
SmellBench evaluates 11 LLM agent setups on 65 architectural smells, finding 47.7% best resolution rate, 63.1% false positives per experts, strong false-positive detection (κ=0.94), but aggressive repairs adding up to...
-
SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair
SmellBench is the first benchmark showing LLM agents resolve 47.7% of architectural code smells while accurately spotting false positives, but aggressive repairs often introduce new smells and degrade overall quality.
-
Patterns of Developer Adoption of LLM-Generated Code Refactoring Suggestions
Analysis of GitHub commits shows developers mostly accept LLM refactoring suggestions without changes, with modifications clustering into five patterns based on activity, prompt, and response validity.
-
Foundation Models as Oracles for Refactoring Correctness Detection
Foundation models serve as effective oracles for detecting refactoring correctness issues in Java programs, achieving up to 93.8% accuracy in zero-shot evaluations on 226 real bugs.
-
Structural Anchors and Reasoning Fragility:Understanding CoT Robustness in LLM4Code
CoT prompting in LLM4Code shows mixed robustness that depends on model family, task structure, and perturbations destabilizing structural anchors, leading to trajectory deformations like lengthening, branching, and si...
-
Your Build Scripts Stink: The State of Code Smells in Build Scripts
The study identifies 13 categories of code smells in build scripts, detects 10,895 occurrences across 5882 scripts from 4877 repositories, and finds common patterns like insecure URLs in Maven and hardcoded paths in G...
-
AI-Generated Smells: An Analysis of Code and Architecture in LLM and Agent-Driven Development
More capable LLMs and agents generate code with greater volume and architectural decay, following a Volume-Quality Inverse Law that neither functional correctness nor prompting mitigates.
-
Qiskit Code Migration with LLMs
A taxonomy-guided RAG system with LLMs reduces hallucinations and improves migration suggestions for Qiskit code compared to unconstrained retrieval.
-
Foundation Models as Oracles for Refactoring Correctness Detection
Foundation models achieve up to 93.8% accuracy detecting refactoring bugs across 47 types in Java IDEs via zero-shot prompting on 226 real cases.
-
A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards
Survey mapping LLM applications in software quality assurance to established standards including ISO/IEC 12207, ISO 25010, CMMI, and TMM, with case studies, challenges, and future directions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.