Pith. sign in

REVIEW 4 major objections 9 minor 66 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Agentic Code Review Closes the Loop on AI Patch Generation

2026-07-08 17:30 UTC pith:SKQSRH34

load-bearing objection Solid system paper on agentic code review for AI-generated PRs. The main experimental design has one real gap — no blind-retry control at benchmark scale — but the paper provides enough partial controls to keep the central claim standing. the 4 major comments →

arxiv 2607.06065 v1 pith:SKQSRH34 submitted 2026-07-07 cs.SE

SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review

Ruoyu Wang , Jierun Chen , Shaowei Wang , Chaofan Tao , Sidi Yang , Yuxin Jiang , Kim-Hui Yap , Lifeng Shang
show 2 more authors
Xiaohui Li Haoli Bai
This is my paper
classification cs.SE
keywords reviewagenticrevisioncodeissuelooptextbfagents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that AI-generated code patches should not be treated as one-shot outputs but as candidates in a closed generate-review-revise loop, where an agentic reviewer actively explores the repository, traces root causes, and produces structured diagnoses that guide revision. The central object is the agentic reviewer: a code review agent that, unlike single-turn reviewers operating on a fixed diff, navigates the repository, runs reproducer scripts, follows call chains across files, and then commits to an approve-or-reject decision with an actionable diagnosis. The paper claims this loop substantially raises resolve rates on SWE-bench Verified — from 27.5% to 56.9% for a weaker generator, from 50.9% to 68.8% for a mid-range one, and from 72.2% to 75.4% for a strong one — and that agentic review outperforms single-turn fixed-context review in both decision accuracy and downstream revision usefulness, with the largest gains on harder patches requiring non-local reasoning. The paper further claims that review trajectories can be distilled into smaller open-weight models, that mixing review and issue-resolution training data improves a single model at both roles, and that review-guided iterative revision is more token-efficient than verifier-selected best-of-N sampling at test time.

Core claim

The paper's central discovery is that the value of code review for AI-generated patches comes not from having more context but from adaptively gathering the right evidence — tracing call chains, running reproducers, comparing implementations — before committing to a judgment. This adaptive exploration lets the reviewer catch symptom fixes that single-turn reviewers approve, and the structured diagnoses it produces carry actionable signal beyond a binary accept/reject: in a controlled ablation, providing the full diagnosis raised post-revision resolve rate to 21% compared to 8% for decision-only feedback and 3% for no feedback, recovering about two-thirds of the gap to an oracle reviewer with

What carries the argument

The generate-review-revise loop, the agentic reviewer with structured JSON output (decision + diagnosis), SWE-Review-Bench (1,384 candidate PRs from 500 SWE-bench Verified issues across three generators), SWE-Review-Traj (8,914 decision-correct review trajectories), three evaluation metrics (Completion Rate, Decision Accuracy, Resolve Rate after Revision), and the mixed-training regime where a single model serves as both patch generator and reviewer.

Load-bearing premise

The paper's Resolve Rate after Revision (RRR) metric assumes that improvements after revision come from the quality of the reviewer's diagnosis, but the revision agent and reviewer share the same scaffold and model family in several experiments, and the main benchmark results do not independently control for the number of revision attempts or the revision agent's own capability apart from the review feedback content.

What would settle it

If a control experiment showed that simply giving the revision agent an additional attempt with the original issue and patch — without any review feedback — produces comparable resolve-rate gains to the full review-guided revision, then the paper's claim that the structured diagnosis carries the improvement would be substantially weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • AI coding systems could shift from one-shot patch submission to iterative closed-loop pipelines where review is not a post-hoc gate but a core mechanism for improving patch quality before merge.
  • The finding that review trajectories transfer to better issue-resolution models suggests that judging and fixing code share a common repository-reasoning skill, which could unify training pipelines for coding agents.
  • Review-guided iterative revision being 6.5x more token-efficient than independent resampling implies that directed feedback is a more cost-effective test-time scaling strategy than brute-force sampling for software engineering tasks.
  • The error analysis showing that false approvals concentrate on subtle logic bugs and insufficient reproducer coverage, while false rejections stem from code misreading, suggests concrete directions: stronger test-adequacy analysis to reduce false approvals and functional-equivalence checks to reduce false rejections.
  • The observation that exploration efficiency rather than volume predicts review quality — with stronger models using fewer steps and tokens yet achieving higher accuracy — implies that reviewer training should focus on targeted evidence-gathering strategies rather than exhaustive search.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. This paper introduces SWE-Review, a framework for closing the issue-resolution loop via agentic code review. The reviewer agent explores a repository, decides whether to accept or request changes on an AI-generated PR, and produces structured diagnoses to guide revision. The authors construct SWE-Review-Bench (1,384 PRs from 500 SWE-bench Verified issues across three generators) and SWE-Review-Traj (8,914 decision-correct review trajectories). They propose three metrics—Completion Rate, Decision Accuracy, and Resolve Rate after Revision (RRR)—and demonstrate that agentic review outperforms single-turn fixed-context review, that review trajectories improve both reviewer distillation and issue-resolution training, and that review-guided iterative revision enables efficient test-time scaling. The work is well-motivated, the benchmark and trajectory dataset are valuable contributions, and the experimental design is generally thorough.

Significance. The paper makes several concrete contributions: (1) a reproducible benchmark (SWE-Review-Bench) with executable verification of both review decisions and downstream revision utility, (2) a publicly released trajectory dataset (SWE-Review-Traj) for open reviewer training, (3) a falsifiable metric (RRR) grounded in executable test outcomes rather than text overlap, and (4) a functional validation ablation (Table 1b) that partially isolates the value of diagnostic content from the binary decision signal. The test-time scaling comparison (Figure 5) and the mixed-training results (Table 3) are informative. The case study (Figure 2, Appendix D) effectively illustrates the agentic-vs-single-turn distinction. These are practical contributions to the SWE agent research community.

major comments (4)
  1. The RRR metric as used in the main benchmark results (Figure 1, Figure 3, Table 2) compares 'no review' (patches kept unchanged, zero revision attempts) against 'review + revision' (rejected patches get one revision with feedback). This conflates two factors: (a) the value of the review feedback content, and (b) the value of simply giving the agent another attempt. Table 1b provides a partial control on 100 correctly-rejected instances (no-review RRR=3%, decision-only=8%, teacher review=21%, oracle=32%), but this is a small sample conditioned on correct rejections (selection bias) and uses a single model configuration. The headline gains in Figure 1 (e.g., 27.5%→56.9% for Qwen3-30B-A3B) could be substantially attributable to the revision attempt itself rather than to the quality of the reviewer's diagnosis. A 'blind retry' control at scale—where rejected patches get a second attempt with
  2. without review feedback—would be needed to fully support the central claim that 'agentic review continually improves PRs through the generate-review-revise loop.' Figure 5 provides indirect evidence (review-guided iterative revision at 38.4% vs. reviewer-gated resampling at 32.3%), but reviewer-gated resampling still uses the reviewer as a gate, so it is not a pure no-feedback control. The paper should either add this control at scale or explicitly acknowledge this confound in the main results discussion and temper the headline claims accordingly.
  3. Section 3.2, Figure 4: The post-hoc difficulty stratification uses file-level divergence from the golden patch and the number of fail-to-pass tests. This criterion is defined post-hoc (i.e., using information not available at review time), and it is correlated with the outcome being measured. The paper should clarify whether the difficulty tertiles are defined over the full benchmark or per-generator split, and should acknowledge that this stratification is descriptive rather than predictive. The claim that 'the gap between agentic and single-turn review widens monotonically from easy to hard patches' is load-bearing for the argument that repository exploration matters most on non-local bugs, but the stratification criterion itself involves golden-patch information, which could introduce circularity.
  4. Table 2: The RRR metric for the 'No Review' baseline equals the original resolve rate (all patches kept unchanged), while the reviewed settings allow one revision for rejected patches. However, the revision agent and the reviewer share the same scaffold and model family in several experiments (e.g., GLM-5 as both reviewer and implicit revision agent in some configurations). The paper does not clearly state which model serves as the revision agent in each cell of Table 2. This information is critical for interpreting RRR, since a stronger revision agent could inflate RRR independently of review quality. The paper should specify the revision agent for each experiment.
minor comments (9)
  1. The abstract states 'continuously improves PRs' but Figure 1 shows results after one review-revision round for the main benchmark. The word 'continuously' is more precisely supported by Figure 5 (up to 4 revision rounds) and Table 4 (up to 5 samples). Consider clarifying which results support the 'continuous' claim.
  2. Section 3.1: The RRR definition says 'patches marked request-changes are returned to the original PR generator with the review feedback for one revision.' It would help to state the revision round cap explicitly in the main text (it appears only in Appendix references).
  3. Table 1a: The semantic validation scores are described as 'mean scores exceed 3.0 on a 5-point scale,' but the overall scores are 3.06 and 3.09. This is barely above 3.0; the phrasing 'exceed 3.0' is technically accurate but could be more precise about the magnitude.
  4. Figure 3: The y-axis label in panel (a) says 'Score (%)' but the metric is Decision Accuracy. Consider labeling it 'DA (%)' for consistency with the text.
  5. Section 4.1: The trajectory filtering retains 8,914 out of 14,156 trajectories based on decision correctness. The paper should report the decision accuracy of the teacher (GLM-5) on the 14,156 PRs to contextualize the filtering rate (63%).
  6. Table 3: The 'RR' column for 'Issue-resolution 1k' is 27.6, but the 'RRR' column is also 27.6. It would help to clarify that RR and RRR are identical when no review is applied (as in the issue-resolution-only rows), to avoid confusion.
  7. Appendix B: Table 5 reports 'Avg total tokens' for Opus 4.6 as 148K, but the text notes that API-billed counts omit reasoning tokens. This caveat should appear in the table caption or as a footnote, not only in the surrounding text, since the comparison with distilled models (2.09M–2.36M) could otherwise be misleading.
  8. The paper uses 'GLM-5' and 'GLM-5 3' (footnote 3) interchangeably. Consider standardizing.
  9. References: Several entries reference 2026 dates (e.g., [1], [4], [24], [35], [43]). If these are genuinely 2026 publications, this is fine; if these are forward-dated preprints, the authors should verify.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive report. The four major comments all identify legitimate gaps in the manuscript that we will address in revision. Specifically: (1) we will add a large-scale blind-retry control to separate the value of review feedback from the value of a second attempt, and will temper headline claims where the confound remains; (2) we will clarify that the difficulty stratification is defined over the full benchmark (not per-generator), is descriptive rather than predictive, and uses golden-patch information unavailable at review time; (3) we will specify the revision agent for every cell of Table 2; and (4) we will add the blind-retry control at scale or explicitly acknowledge the confound. We agree with all four points and will revise accordingly.

read point-by-point responses

  1. Referee: RRR metric conflates value of review feedback content with value of simply giving the agent another attempt; Table 1b is a small sample with selection bias; headline gains could be attributable to revision attempt itself rather than reviewer diagnosis quality; need blind retry control at scale or temper claims.

    Authors: The referee is correct that the main RRR comparisons in Figures 1 and 3 and Table 2 conflate two factors: (a) the value of the reviewer's diagnostic feedback and (b) the value of granting a second revision attempt. Table 1b provides a partial control on 100 correctly-rejected instances (no-review RRR=3%, decision-only=8%, teacher review=21%, oracle=32%), but as the referee notes, this sample is small, conditioned on correct rejections (introducing selection bias), and uses a single model configuration. We agree that a large-scale blind-retry control—where rejected patches receive a second attempt without review feedback—is needed to fully isolate the contribution of the reviewer's diagnosis. We will add this control at scale across all three PR-generator splits in the revised manuscript. Specifically, we will run the revision agent on all rejected patches with only the information that the patch was rejected (matching our existing 'decision-only' condition E.2 but applied to the full benchmark), and report this as a baseline alongside the full-review RRR. This will allow readers to see precisely how much of the headline gain (e.g., 27.5% to 56.9% for Qwen3-30B-A3B) is attributable to the diagnostic content versus the retry itself. We note that Table 1b already provides suggestive evidence that the diagnosis carries substantial value beyond the retry signal (8% for decision-only vs. 21% for teacher review), but we agree this needs to be validated at scale. Regarding Figure 5, the referee correctly observes that reviewer-gated resampling still uses the reviewer as a gate and is therefore not a pure no-feedback control. We will clarify this distinction explicitly in the text and will frame Figure 5 as evidence about the relative efficiency of feedback-guided revision vs.独立 revision: no

  2. Referee: Section 3.2, Figure 4: post-hoc difficulty stratification uses file-level divergence from golden patch and number of fail-to-pass tests; defined post-hoc using information not available at review time; correlated with outcome being measured; should clarify whether difficulty tertiles are defined over full benchmark or per-generator split; should acknowledge stratification is descriptive rather than predictive; claim that gap widens monotonically is load-bearing but could involve circularity from golden-patch information.

    Authors: We agree with all three points raised here. First, the difficulty tertiles are defined over the full 1,384-instance benchmark, not per-generator split. We will state this explicitly in the revised text. Second, we agree that the stratification is descriptive rather than predictive: it uses golden-patch information (file-level divergence and number of fail-to-pass tests) that is not available at review time, and it is correlated with the outcome being measured. We will add a clear acknowledgment that this stratification is intended as a post-hoc descriptive analysis to characterize where agentic review provides the largest benefit, not as a tool that could be applied at inference time. Third, regarding the potential circularity concern: the difficulty criterion does not directly use the review outcome (approve/reject correctness or RRR); it uses properties of the golden patch and the issue's test suite. However, we agree that golden-patch file divergence could correlate with properties that also affect review difficulty, introducing indirect circularity. We will acknowledge this limitation explicitly. We will also temper the claim about monotonic widening: rather than stating it as a definitive finding, we will frame it as an observed trend consistent with our hypothesis that repository exploration matters most on non-local bugs, while noting the post-hoc nature of the stratification as a caveat. We do not believe the circularity is severe enough to invalidate the trend—the stratification is based on structural properties of the correct fix, not on review outcomes—but we agree it should be transparently disclosed. revision: yes

  3. Referee: Table 2: revision agent and reviewer share same scaffold and model family in several experiments; paper does not clearly state which model serves as revision agent in each cell; this information is critical for interpreting RRR since stronger revision agent could inflate RRR independently of review quality; should specify revision agent for each experiment.

    Authors: The referee is correct that the identity of the revision agent is critical for interpreting RRR and that the current manuscript does not specify it clearly for each cell of Table 2. We will add a clear specification of the revision agent for every experiment in the revised manuscript. To preview the setup: in Table 2, the revision agent is the same model that generated the original PR—that is, for the GLM-5 split, GLM-5 serves as both the PR generator and the revision agent; for the Qwen3-Coder-30B-A3B split, Qwen3-Coder-30B-A3B serves as both; and for the Qwen3-30B-A3B split, Qwen3-30B-A3B serves as both. The reviewer model varies by row (Claude Opus 4.6, GLM-5, SWE-Review-8B, SWE-Review-30B-A3B, etc.), but the revision agent is held constant within each generator split. This design ensures that RRR gains are attributable to the quality of the review feedback rather than to a stronger revision agent. We will make this explicit in the table caption and in the experimental setup text. We will also clarify the revision agent for Figures 1, 3, and 5 and for Tables 1b and 3. We agree that without this information, the reader cannot rule out the alternative explanation that a stronger revision agent inflates RRR. revision: yes

  4. Referee: Figure 5 provides indirect evidence (review-guided iterative revision at 38.4% vs. reviewer-gated resampling at 32.3%), but reviewer-gated resampling still uses the reviewer as a gate, so it is not a pure no-feedback control.

    Authors: The referee is correct that reviewer-gated resampling is not a pure no-feedback control because it still uses the reviewer's approval decision as a gate for early stopping. The comparison in Figure 5 therefore isolates the value of diagnostic feedback for guiding revision (versus using the reviewer only as a binary gate), but it does not isolate the value of review feedback from the value of a second attempt more broadly. We will clarify this in the revised text: Figure 5 addresses a different question—whether structured diagnostic feedback is more effective than binary gating for test-time scaling—rather than the question of whether review feedback adds value over a blind retry. The latter question is addressed by the blind-retry control we will add in response to the first major comment. We will also add cross-references between Figure 5 and the new blind-retry control to make the logical structure clear: the blind-retry control isolates feedback value vs. retry value, while Figure 5 isolates feedback value vs. gate-only value within a test-time scaling setting. We agree that the current text does not adequately distinguish these two questions. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; one non-load-bearing self-citation.

full rationale

The paper's central claims are grounded in external benchmarks (SWE-bench Verified with executable test suites) and operational metrics defined against executable verification of patch correctness. Decision Accuracy (DA) compares the reviewer's approve/request-changes decision against the patch's true resolve status (external ground truth). Resolve Rate after Revision (RRR) measures final resolve rate after a standardized revision procedure, again verified by executable tests. The SWE-Review-Traj training dataset is filtered by decision correctness against executable patch outcomes (Section 4.1: 'retaining trajectories where the reviewer correctly approves a resolving patch or correctly requests changes on a non-resolving one'), which is external validation, not a self-referential loop. The main results (Figure 1, Figure 3, Table 2) compare against external baselines: no-review, single-turn review, and frontier models. The one self-citation is reference [35] (SWE-Lego-Verifier), which shares several authors with the present paper and is used as a verifier baseline in Figure 5. However, this citation is not load-bearing for any central claim — it serves as a comparison point that the paper's reviewer-based approach outperforms, and the results do not depend on its correctness. The reader's concern about RRR conflating review feedback value with revision-attempt value is a valid experimental design concern (correctness risk), but it is not circularity: the metric is defined operationally and measured against an external standard, and the paper provides a partial control (Table 1b) that isolates feedback content from the revision signal. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new postulated entities (particles, forces, dimensions). The free parameters are experimental design choices (filtering thresholds, iteration caps, difficulty criteria) rather than fitted model constants. The axioms are standard domain assumptions for SWE-bench-based research: test-suite correctness as ground truth, benchmark representativeness, and scaffold consistency.

free parameters (4)
  • Difficulty stratification criterion = file-level divergence + fail-to-pass test count tertiles
    Post-hoc criterion using golden patch divergence, which is not available at inference time. Used in Section 3.2 and Figure 4 to stratify results.
  • Trajectory filtering threshold (decision correctness) = 8,914 of 14,156 trajectories retained
    Filtering rate selected to retain correct decisions; affects training data quality.
  • Revision rounds cap = 4 revision rounds (5 total samples)
    Chosen for the iterative loop; affects RRR and test-time scaling comparisons.
  • Max patch size filter = 50 KB
    Ad hoc threshold for filtering oversized patches in trajectory curation (Section 4.1).
axioms (4)
  • domain assumption SWE-bench Verified issues with executable test suites are a representative sample of real-world software issues
    The entire benchmark and evaluation framework depends on this assumption. Section 3.1 constructs SWE-Review-Bench from 500 SWE-bench Verified issues.
  • domain assumption A patch's resolve status (passing FAIL_TO_PASS and PASS_TO_PASS tests) is a valid ground truth for review correctness
    DA and RRR metrics are defined against executable verification of patch correctness (Section 3.1). This assumes test suites are sufficient to determine correctness.
  • domain assumption The OpenHands-SDK scaffold provides a fair and consistent agent environment across all experiments
    All PR generation and review experiments use this scaffold (Sections 3.1, 4.1). Results may not transfer to other scaffolds.
  • domain assumption LLM-as-judge scores from Claude Opus 4.6 and GPT-5.4 are reliable for semantic validation of review diagnoses
    Table 1a uses these judges to validate diagnosis quality. Cohen's kappa=0.72 is cited as evidence of agreement, but the scores themselves are modest (3.0/5 mean).

pith-pipeline@v1.1.0-glm · 32690 in / 3019 out tokens · 400513 ms · 2026-07-08T17:30:44.354959+00:00 · methodology

0 comments
read the original abstract

Coding agents increasingly generate pull requests (PRs) for real-world software issues, yet one-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision. We introduce \textbf{SWE-Review}, a framework for closing this loop with agentic code review. Given an issue and an AI-generated PR, a reviewer agent explores the repository, decides whether the PR should be accepted, and provides structured feedback for revision. We evaluate this setting with our proposed \textbf{SWE-Review-Bench} to measure both review correctness and downstream revision usefulness. We further curate \textbf{SWE-Review-Traj} dataset to study broader applications of agentic review and fill the data-scarcity gap for open reviewer training. Experiments show that agentic review continuously improves PRs through a generate-review-revise loop, outperforms single-turn fixed-context review in both decision accuracy and resolve rate after revision, transfers beyond review to improve issue-resolution models, and enables effective and efficient test-time scaling. These results position agentic code review as a practical mechanism for moving AI coding agents from one-shot PR generation toward closed-loop issue resolution.

Figures

Figures reproduced from arXiv: 2607.06065 by Chaofan Tao, Haoli Bai, Jierun Chen, Kim-Hui Yap, Lifeng Shang, Ruoyu Wang, Shaowei Wang, Sidi Yang, Xiaohui Li, Yuxin Jiang.

Figure 1
Figure 1. Figure 1: SWE-Review turns one-shot PR generation into closed-loop issue resolution. AI cod￾ing agents generate candidate PRs from real software issues; an agentic reviewer then inspects the repository, issue, and candidate PR in an executable environment before emitting a binary decision and structured diagnosis. Approved PRs are accepted as resolved, while rejected PRs are returned to the coding agent for review-g… view at source ↗
Figure 2
Figure 2. Figure 2: Agentic review traces the bug across modules to its upstream root cause; single-turn approves a downstream symptom fix that silently returns wrong results. On sympy-13877, the candidate PR adds a NaN guard in exprtools.py that prevents the crash but still leaves det() returning incorrect nan values. The agentic reviewer explores the repository, follows the call chain to matrices.py, identifies the discarde… view at source ↗
Figure 3
Figure 3. Figure 3: Agentic review produces more accurate approve/request-changes decisions and more useful diagnoses that lead to higher resolve rates after revision than single-turn review. PRs are generated by three models of varying capability (GLM-5, Qwen3-Coder-30B-A3B, and Qwen3- 30B-A3B). All settings use the same reviewer model (Claude Opus 4.6 [4]), and completion rates are generally above 95% across settings. Easy … view at source ↗
Figure 4
Figure 4. Figure 4: Agentic review increasingly outperforms single-turn review as difficulty rises. We stratify SWE-Review-Bench into easy, medium, and hard tertiles using the post-hoc difficulty crite￾rion described in Section 3.2. n=422). This stratification studies reviewer behavior under diverse candidate qualities rather than a single generator prior. 3.2 Agentic Review Better Guides PR Revision Than Single-Turn Review W… view at source ↗
Figure 5
Figure 5. Figure 5: Agentic review enables effective and efficient test-time scaling for issue resolution. All methods use Qwen3-30B-A3B as the issue-resolution model and compare size-matched 8B re￾viewer/verifier judges; both panels use a log-scaled sample-budget x-axis. Review-guided iterative revision achieves the best resolve rate while using far fewer samples on average than independent resampling or verifier-selected be… view at source ↗
Figure 6
Figure 6. Figure 6: Test-time scaling with Qwen3-30B-A3B as the coding agent. Reviewer dominates Verifier and Critic across all budgets, while using far fewer samples due to early stopping. 1. Oracle: reports Pass@K, i.e., the fraction of instances for which at least one of the K samples resolves the issue, serving as an upper bound on any selector. 2. Reviewer: our trained SWE-Review-8B used in a reviewer-gated resampling pr… view at source ↗
Figure 7
Figure 7. Figure 7: Test-time scaling with Qwen3-Coder-30B-A3B as the coding agent. At K=16, Re￾viewer reaches 48.1% vs. 44.4% for Verifier while using only 6.0 samples on average (62.5% reduc￾tion). B Extended Analysis of Reviewer Behavior This section provides extended analyses of the agentic review pipeline, examining reviewer explo￾ration behavior (Section B.1), failure modes (Section B.2), and token consumption (Section … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 66 canonical work pages · 15 internal anchors

  1. [2]

    URLhttps://arxiv.org/abs/2602.15763

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [3]

    Learning From Mistakes Makes LLM Better Reasoner

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [4]

    Claude code by anthropic.https://www.anthropic.com/product/ claude-code, 2026

    Anthropic. Claude code by anthropic.https://www.anthropic.com/product/ claude-code, 2026

    work page 2026

  4. [5]

    Introducing claude opus 4.6.https://www.anthropic.com/news/ claude-opus-4-6, 2026

    Anthropic. Introducing claude opus 4.6.https://www.anthropic.com/news/ claude-opus-4-6, 2026

    work page 2026

  5. [6]

    Cursor: The Best Way to Code with AI.https://cursor.com/, 2026

    Anysphere, Inc. Cursor: The Best Way to Code with AI.https://cursor.com/, 2026. Accessed: 2026-05-07

    work page 2026

  6. [7]

    Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents

    Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. arXiv preprint arXiv:2505.20411, 2025

    work page arXiv 2025

  7. [8]

    Introducing devin, the first ai software engineer.https://cognition.ai/ blog/introducing-devin, 2024

    Cognition AI. Introducing devin, the first ai software engineer.https://cognition.ai/ blog/introducing-devin, 2024

    work page 2024

  8. [9]

    SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long- horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [10]

    Re- solving code review comments with machine learning

    Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, et al. Re- solving code review comments with machine learning. InProceedings of the 46th international conference on software engineering: software engineering in practice, pages 204–215, 2024

    work page 2024

  10. [11]

    Aider: Ai pair programming in your terminal.https://github.com/ paul-gauthier/aider, 2023

    Paul Gauthier. Aider: Ai pair programming in your terminal.https://github.com/ paul-gauthier/aider, 2023. GitHub repository

    work page 2023

  11. [12]

    Codefuse-cr-bench: A comprehensiveness-aware benchmark for end-to-end code review evaluation in python projects

    Hanyang Guo, Xunjin Zheng, Zihan Liao, Hang Yu, Peng Di, Ziyin Zhang, and Hong-Ning Dai. Codefuse-cr-bench: A comprehensiveness-aware benchmark for end-to-end code review evaluation in python projects. arXiv preprint arXiv:2509.14856, 2025

    work page arXiv 2025

  12. [13]

    R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents

    Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2e- gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents.arXiv preprint arXiv:2504.07164, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [14]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [15]

    Deepreview: automatic code review using deep multi-instance learning

    Heng-Yi Li, Shu-Ting Shi, Ferdian Thung, Xuan Huo, Bowen Xu, Ming Li, and David Lo. Deepreview: automatic code review using deep multi-instance learning. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 318–330. Springer, 2019. 10

    work page 2019

  15. [16]

    Auger: automatically generating review comments with pre-training models

    Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. Auger: automatically generating review comments with pre-training models. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1009–1021, 2022

    work page 2022

  16. [17]

    Automating Code Review Activities by Large-Scale Pre-training

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al. Codereviewer: Pre-training for automating code review activities. arXiv preprint arXiv:2203.09095, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  17. [18]

    Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), pages 647–658. IEEE, 2023

    work page 2023

  18. [19]

    Self-refine: Iterative re- finement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative re- finement with self-feedback. Advances in neural information processing systems, 36:46534– 46594, 2023

    work page 2023

  19. [20]

    Generative Reward Models

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castri- cato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.12832, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [21]

    Crscore: Grounding auto- mated evaluation of code review comments in code claims and smells

    Atharva Naik, Marcus Alenius, Daniel Fried, and Carolyn Rose. Crscore: Grounding auto- mated evaluation of code review comments in code claims and smells. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume1: Long Papers), pages 9049–9076, 2025

    work page 2025

  21. [22]

    Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study

    Doriane Olewicki, Leuson Da Silva, Suhaib Mujahid, Arezou Amini, Benjamin Mah, Marco Castelluccio, Sarra Habchi, Foutse Khomh, and Bram Adams. Impact of llm-based review comment generation in practice: A mixed open-/closed-source user study. arXiv preprint arXiv:2411.07091, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [23]

    Introducing swe-bench verified.https://openai.com/index/ introducing-swe-bench-verified/, August 2024

    OpenAI. Introducing swe-bench verified.https://openai.com/index/ introducing-swe-bench-verified/, August 2024. Blog post

    work page 2024

  23. [24]

    Introducing codex.https://openai.com/zh-Hans-CN/index/ introducing-codex/, 2026

    OpenAI. Introducing codex.https://openai.com/zh-Hans-CN/index/ introducing-codex/, 2026

    work page 2026

  24. [25]

    Introducing gpt-5.4.https://openai.com/zh-Hans-CN/index/ introducing-gpt-5-4/, 2026

    OpenAI. Introducing gpt-5.4.https://openai.com/zh-Hans-CN/index/ introducing-gpt-5-4/, 2026

    work page 2026

  25. [26]

    Opencode.https://github.com/opencode-ai/opencode, 2025

    OpenCode. Opencode.https://github.com/opencode-ai/opencode, 2025

    work page 2025

  26. [27]

    Training Software Engineering Agents and Verifiers with SWE-Gym

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [28]

    Cr-bench: Evaluating the real-world utility of ai code review agents, 2026

    Kristen Pereira, Neelabh Sinha, Rajat Ghosh, and Debojyoti Dutta. Cr-bench: Evaluating the real-world utility of ai code review agents, 2026

    work page 2026

  28. [29]

    Pr-agent: Ai-powered pull request analysis and feedback.https://github.com/ The-PR-Agent/pr-agent, 2023

    Qodo. Pr-agent: Ai-powered pull request analysis and feedback.https://github.com/ The-PR-Agent/pr-agent, 2023. GitHub repository

    work page 2023

  29. [30]

    Qwen3-coder: Agentic coding in the world.https://qwenlm.github.io/blog/ qwen3-coder/, 2025

    Qwen. Qwen3-coder: Agentic coding in the world.https://qwenlm.github.io/blog/ qwen3-coder/, 2025

    work page 2025

  30. [31]

    Automatic code review by learning the revision of source code, 2019

    Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. Automatic code review by learning the revision of source code, 2019. 11

    work page 2019

  31. [32]

    Swe-rm: Execution-free feedback for software engineering agents

    KaShun Shum, Binyuan Hui, Jiawei Chen, Lei Zhang, Jiaxi Yang, Yuzhen Huang, Junyang Lin, Junxian He, et al. Swe-rm: Execution-free feedback for software engineering agents. arXiv preprint arXiv:2512.21919, 2025

    work page arXiv 2025

  32. [33]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [34]

    Bitsai-cr: Automated code review via llm in practice

    Tao Sun, Jian Xu, Yuanpeng Li, Zhao Yan, Ge Zhang, Lintao Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, et al. Bitsai-cr: Automated code review via llm in practice. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 274–285, 2025

    work page 2025

  34. [35]

    Codeagent: Autonomous communicative agents for code review

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F Bissyandé. Codeagent: Autonomous communicative agents for code review. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11279–11313, 2024

    work page 2024

  35. [36]

    Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving

    Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, et al. Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving. arXiv preprint arXiv:2601.01426, 2026

    work page arXiv 2026

  36. [37]

    Towards automating code review activities, 2021

    Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. Towards automating code review activities, 2021

    work page 2021

  37. [38]

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [39]

    Learning to verify ai-generated code.https: //openhands.dev/blog/20260305-learning-to-verify-ai-generated-code, March 2026

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Learning to verify ai-generated code.https: //openhands.dev/blog/20260305-learning-to-verify-ai-generated-code, March 2026. Blog post

    work page arXiv 2026

  39. [40]

    Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

    Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning to critique is more effective than learning to imitate. arXiv preprint arXiv:2501.17703, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [41]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [42]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated soft- ware engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    work page 2024

  42. [43]

    SWE-smith: Scaling Data for Software Engineering Agents

    John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [44]

    Aacr-bench: Evaluating automatic code review with holistic repository-level context

    Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, et al. Aacr-bench: Evaluating automatic code review with holistic repository-level context. arXiv preprint arXiv:2601.19494, 2026

    work page arXiv 2026

  44. [45]

    SWE-bench Goes Live!

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  45. [46]

    Autocoderover: Au- tonomous program improvement

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Au- tonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592–1604, 2024. 12

    work page 2024

  46. [47]

    Code review agent benchmark, 2026

    Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, and Abhik Roychoudhury. Code review agent benchmark, 2026

    work page 2026

  47. [48]

    Issue-resolution 3k+Review 3k

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595– 46623, 2023. A Additional Test-Time Scaling Results Section 4.3 focuses on three test-time scaling stra...

    work page 2023

  48. [49]

    2.tokenizer.apply_chat_template()renders each trajectory into the model’s native chat for- mat with correct special tokens

    Raw agent trajectories are cleaned and standardized. 2.tokenizer.apply_chat_template()renders each trajectory into the model’s native chat for- mat with correct special tokens

    work page

  49. [50]

    Rendered text is split by turn; error-preceding assistant turns are flagged for loss masking

    work page

  50. [51]

    <pyshell#4>

    Final output is ShareGPT JSON with per-turn mask annotations. The tokenizer must exactly match the target base model. D Case Study: sympy-13877 This appendix expands the case study referenced in Section 3.2 and Figure 2. Overview.The case compares the same candidate PR under two review settings: single-turn Opus with diff+context and agentic Opus. The can...

    work page 2041

  51. [52]

    **What does the user expect?** (exact: no error, specific output, specific return value)

    work page

  52. [53]

    import json; open(’/tmp/p.diff’,’w’).write(json.load(open(’/review_data/predicted.json ’))[’patch’])

    **What actually happens?** (exact: error type + message, wrong output, wrong value) Do NOT read ‘predicted.json‘ yet. ### Step 2: Find the Root Cause Go to ‘/testbed‘. Trace from the entry point down through the call stack: - Where is the bug triggered? (What does the user call?) - Follow the chain: entry -> function -> callee -> callee’s callee... 26 - *...

    work page

  53. [54]

    **Bug reproduction test**: the reported case now works

    work page

  54. [55]

    no exception

    **Full output/behavior check** -- not just "no exception": - If expected: "no output" -> use ‘subprocess.run([...], capture_output=True)‘ and assert ‘ result.stdout == ""‘ - If expected: specific return value -> assert the exact value, not just that no error occurred - If expected: no crash -> verify the result is semantically correct too

    work page

  55. [56]

    import json; r=json.load(open(’/review_report.json’)); print(’OK:’, r[’decision’][’ recommendation’])

    **Regression test**: existing behavior still works If any test fails -> ‘request_changes‘. > **If a test fails: IMMEDIATELY go to Step 6 and write ‘request_changes‘. Do NOT investigate why the test failed or keep exploring. The failure itself is the defect.** 27 ### Step 6: Write Report > **Budget reminder**: You have 100 iterations total. By this step, S...

    work page

  56. [57]

    Tests pass including full output/behavior verification

    work page

  57. [58]

    The patch changes the ROOT CAUSE location identified in Step 2

    work page

  58. [59]

    decision

    No high-severity defects **Request changes** if ANY: - Any test fails (including output verification) - The patch is a symptom fix (Step 4 HARD RULE triggered) - The patch fixes only part of the affected code paths - A ‘high‘ severity defect exists - Empty patch ## Defect Severity Guide **High** (always -> ‘request_changes‘): - Incorrect result/output for...

    work page

  59. [62]

    decision

    PR metadata (title and description) Your task is to review the candidate patch and determine whether it correctly fixes the reported issue. You must output a JSON review report with the following schema: { "decision": { "recommendation": "approve" or "request_changes", "confidence": 0.0 to 1.0 }, "summary": { "problem": "1 sentence: root cause (file + fun...

    work page

  60. [63]

    A bug report (issue description)

    work page

  61. [64]

    A candidate patch (unified diff format)

    work page

  62. [65]

    PR metadata (title and description)

    work page

  63. [66]

    golden patch

    Full source files affected by the patch (pre-change versions) Use the file context to understand the surrounding code, existing patterns, and potential regressions. Your task is to review the candidate patch and determine whether it correctly fixes the reported issue. [Same JSON output schema as Setting 1] 29 E.4 Oracle Review Prompt The oracle review has...

    work page

  64. [67]

    **diagnosis_accuracy** -- Does the review correctly identify the root cause the candidate patch missed? - 5: Pinpoints exact missing/incorrect change; aligns with golden patch - 3: General area right, wrong depth - 1: Unrelated, hallucinated, or wrong target

    work page

  65. [68]

    **suggestion_correctness** -- Do suggestions point toward the change golden patch makes? - 5: Suggestion is semantically equivalent to golden patch - 3: Direction right but details insufficient - 1: Would not fix or introduces new issues

    work page

  66. [69]

    reasoning

    **grounding_quality** -- Are claims verifiable from candidate patch + problem_statement? - 5: Every major claim is grounded - 3: Mostly grounded, some unsupported - 1: Contains hallucinated code paths / invented symptoms ## Output (JSON only) { "reasoning": "<2-5 sentences analysis BEFORE scoring>", "diagnosis_accuracy": {"score": <1-5>, "justification": ...

    work page