REVIEW 22 cited by
SpecRover: Code Intent Extraction via LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Autonomous program improvement typically involves automatically producing bug fixes and feature additions. Such program improvement can be accomplished by a combination of large language model (LLM) and program analysis capabilities, in the form of an LLM agent. Since program repair or program improvement typically requires a specification of intended behavior - specification inference can be useful for producing high quality program patches. In this work, we examine efficient and low-cost workflows for iterative specification inference within an LLM agent. Given a GitHub issue to be resolved in a software project, our goal is to conduct iterative code search accompanied by specification inference - thereby inferring intent from both the project structure and behavior. The intent thus captured is examined by a reviewer agent with the goal of vetting the patches as well as providing a measure of confidence in the vetted patches. Our approach SpecRover (AutoCodeRover-v2) is built on the open-source LLM agent AutoCodeRover. In an evaluation on the full SWE-Bench consisting of 2294 GitHub issues, it shows more than 50% improvement in efficacy over AutoCodeRover. Compared to the open-source agents available, our work shows modest cost ($0.65 per issue) in resolving an average GitHub issue in SWE-Bench lite. The production of explanation by SpecRover allows for a better "signal" to be given to the developer, on when the suggested patches can be accepted with confidence. SpecRover also seeks to demonstrate the continued importance of specification inference in automated program repair, even as program repair technologies enter the LLM era.
Forward citations
Cited by 22 Pith papers
-
NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition
NoCode-bench provides 634 documentation-change-to-code feature addition tasks, and the best evaluated LLM succeeds on only 28.07% of the human-verified subset.
-
Adversarial Reasoning for Repair Based on Inferred Program Intent
AdverIntent-Agent infers multiple adversarial program intents, generates tests for each, and produces patches, correctly repairing 77 Defects4J and 105 HumanEval-Java bugs.
-
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch
WebGen-Bench provides 101 website-building tasks with 647 atomic test cases, and a trained Qwen2.5-Coder-32B model reaches 38.2% accuracy, outperforming DeepSeek-R1's 27.8%.
-
Coupling Planning with Episodic Memory in LLM Agents for Software Issue Resolution
Coupling a hierarchical phase planner with episodic memory lets an LLM repair agent hold onto its diagnosis and recover from failed edits, resolving 5.0pp more SWE-bench Verified issues.
-
SVRepair: Structured Visual Reasoning for Automated Program Repair
A multimodal program-repair system that converts bug screenshots into semantic scene graphs and iteratively crops to bug regions, reporting 36.47% on SWE-Bench M, 38.02% on MMCode, and 95.12% on CodeVision.
-
Git Context Controller: Manage the Context of LLM-based Agents like Git
GCC, a Git-inspired context management layer, is reported to lift a Claude-based agent to 48.00% on SWE-Bench-Lite and to enable a self-replicating CLI that resolves 40.7% of tasks versus 11.7% without it.
-
Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback
Meta's Engineering Agent solved 42.3% of curated internal test-failure tasks offline and landed 25.5% of generated fixes in production over three months.
-
Is Your Automated Software Engineer Trustworthy?
BouncerBench evaluates whether LLM coding agents can abstain from acting on underspecified tickets and incorrect patches, and shows current models rarely abstain correctly.
-
From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs
Adding bug-inducing commit context to LLM prompts raises correct regression repairs from 9 to 16 of 99 Java bugs (1.8x).
-
Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement
AlignMind, a multi-agent system with theory-of-mind helpers, refines software requirements through multi-round dialogue, outperforming a direct-prompt baseline in LLM-judged quality and lexical richness, at high token...
-
OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution
OmniGIRL introduces a 959-instance multilingual, multimodal GitHub issue resolution benchmark; the best result, GPT-4o with Agentless-X, resolves only 8.6% of issues.
-
kAgent: An execution-guided crash resolution agent for the Linux kernel
An LLM agent with execution-trace-guided hypothesis generation resolves about 65% of fuzzed Linux kernel crashes when told which file to fix, but most of its 'fixed' crashes are resolved by deleting or bypassing code.
-
PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification
A rule-based LLM patching agent with a refinement loop resolves 45.3% and 53.6% on SWE-bench Lite and Verified for about $1 per instance, beating open-source baselines.
-
Evaluating Agent-based Program Repair at Google
A SWE-Agent-style AI repaired 73% of machine-reported and 25.6% of human-reported bugs in a curated 178-bug Google-internal benchmark, with 43% and 17.9% matching the human fix.
-
TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?
Introduces TDD-Bench-Verified, a 449-issue benchmark for test-driven development, and Auto-TDD, an LLM pipeline that achieves a higher fail-to-pass rate (up to 23.6% with GPT-4o) than prior methods.
-
Detection Time Distribution Predicted Using Absorbing Boundary Conditions and Imaginary Potentials
Three inequivalent quantum detection-time proposals yield waveguide arrival distributions showing partial reflection, spin-orientation independence (for tested parameters), and width dependence when spin couples to th...
-
Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
On 372 multi-hunk bugs, coding agents repair between 26% and 93% of defects, accuracy falls as edits become more divergent and dispersed, and failed repairs consume up to 343% more tokens.
-
ViTAD: Timing Violation-Aware Debugging of RTL Code using Large Language Models
ViTAD uses a graph of signal timing dependencies plus an LLM to diagnose and repair RTL timing violations, reportedly improving success rate by 19.30 points over a basic LLM baseline.
-
SemAgent: A Semantics Aware Program Repair Agent
SemAgent, a workflow-based repair agent using execution, issue, and code semantics, reports solving 134 of 300 SWE-bench Lite issues (44.66%).
-
CoRet: Improved Retriever for Code Editing
CoRet fine-tunes a code encoder with a repository-level likelihood loss and call-graph context, improving retrieval recall on SWE-bench and Long Code Arena.
-
Self-Evolving Coding Agents
A systematic survey that defines self-evolving coding agents, organizes systems into a five-category object-centered taxonomy, and analyzes when and on what evidence these agents evolve.
-
Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering
A survey-plus-benchmark argues that current code LLMs fail safety thresholds and need stronger governance, but the supporting experiment is incomplete.
Discussion (0). Continue with ORCID to comment.