Pith. sign in

REVIEW 17 cited by

SpecRover: Code Intent Extraction via LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.02232 v4 pith:JO4ZQP6Z submitted 2024-08-05 cs.SE cs.AI

Haifeng Ruan , Yuntong Zhang , Abhik Roychoudhury This is my paper
classification cs.SEcs.AI
keywords programspecificationagentimprovementinferencepatchesspecrovergithub
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Autonomous program improvement typically involves automatically producing bug fixes and feature additions. Such program improvement can be accomplished by a combination of large language model (LLM) and program analysis capabilities, in the form of an LLM agent. Since program repair or program improvement typically requires a specification of intended behavior - specification inference can be useful for producing high quality program patches. In this work, we examine efficient and low-cost workflows for iterative specification inference within an LLM agent. Given a GitHub issue to be resolved in a software project, our goal is to conduct iterative code search accompanied by specification inference - thereby inferring intent from both the project structure and behavior. The intent thus captured is examined by a reviewer agent with the goal of vetting the patches as well as providing a measure of confidence in the vetted patches. Our approach SpecRover (AutoCodeRover-v2) is built on the open-source LLM agent AutoCodeRover. In an evaluation on the full SWE-Bench consisting of 2294 GitHub issues, it shows more than 50% improvement in efficacy over AutoCodeRover. Compared to the open-source agents available, our work shows modest cost ($0.65 per issue) in resolving an average GitHub issue in SWE-Bench lite. The production of explanation by SpecRover allows for a better "signal" to be given to the developer, on when the suggested patches can be accepted with confidence. SpecRover also seeks to demonstrate the continued importance of specification inference in automated program repair, even as program repair technologies enter the LLM era.

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Dynamic analysis enhances issue resolution

    cs.SE 2026-03 conditional novelty 7.0 of 10

    DAIRA integrates dynamic tracing into LLM agents to achieve 79.4% resolution rate on SWE-bench Verified for code defect repair.

  2. Investigating Test Overfitting on SWE-bench

    cs.SE 2025-11 unverdicted novelty 7.0 of 10

    The first empirical study of test overfitting shows that auto-generated tests from issues can lead to code that passes observed tests but misses important cases or breaks functionality in SWE-bench issue resolution.

  3. NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition

    cs.SE 2025-07 conditional novelty 7.0 of 10

    NoCode-bench provides 634 documentation-change-to-code feature addition tasks, and the best evaluated LLM succeeds on only 28.07% of the human-verified subset.

  4. Beyond Fixed Tests: Repository-Level Issue Resolution as Coevolution of Code and Behavioral Constraints

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    Agent-CoEvo is a multi-agent LLM framework that coevolves code patches and test patches to resolve repository-level issues, outperforming fixed-test baselines on SWE-bench Lite and SWT-bench Lite.

  5. Dynamic analysis enhances issue resolution

    cs.SE 2026-03 unverdicted novelty 6.0 of 10

    Embedding dynamic analysis into an LLM repair agent yields a claimed 79.4% resolution rate on SWE-bench Verified while cutting token use by about 25%.

  6. SVRepair: Structured Visual Reasoning for Automated Program Repair

    cs.SE 2026-02 conditional novelty 6.0 of 10

    A multimodal program-repair system that converts bug screenshots into semantic scene graphs and iteratively crops to bug regions, reporting 36.47% on SWE-Bench M, 38.02% on MMCode, and 95.12% on CodeVision.

  7. Git Context Controller: Manage the Context of LLM-based Agents like Git

    cs.SE 2025-07 reject novelty 6.0 of 10

    GCC, a Git-inspired context management layer, is reported to lift a Claude-based agent to 48.00% on SWE-Bench-Lite and to enable a self-replicating CLI that resolves 40.7% of tasks versus 11.7% without it.

  8. Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Meta's Engineering Agent solved 42.3% of curated internal test-failure tasks offline and landed 25.5% of generated fixes in production over three months.

  9. Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios

    cs.SE 2025-03 accept novelty 6.0 of 10

    Empirical study of 3977 agent trajectories finds Python execution errors correlate with lower success rates on GitHub issues, flags challenging errors, and reports three confirmed bugs in the SWE-Bench platform.

  10. Agentless: Demystifying LLM-based Software Engineering Agents

    cs.SE 2024-07 conditional novelty 6.0 of 10

    Agentless, a basic three-phase LLM pipeline for bug localization, repair, and validation, outperforms complex open-source agents on SWE-bench Lite with 32% success rate at $0.70 cost.

  11. Detection Time Distribution Predicted Using Absorbing Boundary Conditions and Imaginary Potentials

    quant-ph 2026-03 unverdicted novelty 5.0 of 10

    Three inequivalent quantum detection-time proposals yield waveguide arrival distributions showing partial reflection, spin-orientation independence (for tested parameters), and width dependence when spin couples to th...

  12. Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

    cs.SE 2026-02 unverdicted novelty 5.0 of 10

    Agent-generated tests mainly act as observational feedback channels and do not meaningfully improve issue resolution success in current LLM software engineering agents.

  13. Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair

    cs.SE 2025-11 conditional novelty 5.0 of 10

    On 372 multi-hunk bugs, coding agents repair between 26% and 93% of defects, accuracy falls as edits become more divergent and dispersed, and failed repairs consume up to 343% more tokens.

  14. ViTAD: Timing Violation-Aware Debugging of RTL Code using Large Language Models

    cs.AR 2025-08 unverdicted novelty 5.0 of 10

    ViTAD uses a graph of signal timing dependencies plus an LLM to diagnose and repair RTL timing violations, reportedly improving success rate by 19.30 points over a basic LLM baseline.

  15. Self-Evolving Coding Agents

    cs.SE 2026-08 conditional novelty 4.0 of 10

    A systematic survey that defines self-evolving coding agents, organizes systems into a five-category object-centered taxonomy, and analyzes when and on what evidence these agents evolve.

  16. LLM-Based Automated Diagnosis Of Integration Test Failures At Google

    cs.SE 2026-04 unverdicted novelty 4.0 of 10

    Auto-Diagnose applies LLMs to summarize and diagnose root causes of integration test failures, reporting 90.14% accuracy on 71 manual cases and positive adoption after Google-wide rollout.

  17. Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering

    cs.SE 2025-08 reject novelty 3.0 of 10

    A survey-plus-benchmark argues that current code LLMs fail safety thresholds and need stronger governance, but the supporting experiment is incomplete.

Pith tools