{"id":"43c97ae6-a312-43b8-b60e-9087211f1d4c","arxiv_id":"2608.08950","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RETRACE improves coding-agent patch correctness by having the model guess the problem a patch solves without seeing the original report, then reconciling that guess with the report to decide whether to submit or revise.","lead":"This paper introduces RETRACE, a training-free add-on that checks a coding agent's patch by reconstructing, from the patch and its trajectory alone, the problem the patch appears to solve, then comparing that reconstruction with the original bug report. On SWE-bench Verified it lifts Pass@1 by 7.0 and 3.6 percentage points for two backbones on mini-SWE-agent, with comparable gains on OpenHands.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The independence of the backward verification signal is asserted, not demonstrated: same-backbone reconstruction from a trajectory generated with the issue in context may simply restate the generator's interpretation, so the Pass@1 gains could come from extra structured inference rather than from…","rationale":"Good-faith reading: RETRACE is a plausible, training-free post-hoc verification loop, and the end-to-end gains on SWE-bench Verified are consistent across two scaffolds and two backbones, which is real evidence of usefulness. The weak point is mechanistic: the paper labels backward reconstruction 'independent' because I is withheld, but every input to that stage was produced by a model that had I in its context, and the verifier is the same model. Under those conditions there is no a priori reason the reconstructed problem is more trustworthy than the generator's own self-review. The existing Self-Refine comparison is informative but not isolating, and the forward-only ablation shows that a substantial part of the gain does not depend on the backward stage at all. The proposed test arms would settle whether withholding I changes predictions: if it does not, the headline claim should be softened to 'structured bidirectional reasoning helps' rather than 'independent verification helps.' Because code, prompts, and seeds are not provided, this cannot be checked post hoc from the paper; the conditional verdict should stand pending the independence control, public artifacts, and error bars.","tokens_in":11570,"tokens_out":6753,"duration_ms":67853,"concrete_test":"On the 120-issue subset, run three backward variants with identical reconciliation logic: (a) the published protocol with I withheld; (b) the same protocol with I appended to the backward reconstruction prompt; (c) a control in which backward reconstruction receives a patch from a different issue while the comparison still uses the original I. Use at least three seeds and report Pass@1 and the distribution of Same/Partial/Different verdicts. If (a) and (b) are indistinguishable, withholding I is not the active ingredient; if (c) produces high alignment rates, the verdict has little discriminative validity and the 'independent signal' explanation is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RETRACE's central claim depends on the backward stage in 'Backward Reconstruction of the Addressed Problem' producing an independent signal by withholding I. The paper does not show this independence. τ_b consists of observations and decisions made while the agent had I available; the patch itself is a product of that same context, so p and τ_b are causally downstream of I. 'Explicit restatements' may be stripped, but the remaining evidence still encodes the generator's interpretation. The same backbone model then performs backward reconstruction, so bI can be a rationalization of the model's own patch rather than an external check. The controls do not isolate the withheld-issue mechanism: Self-Refine differs in prompt structure, evidence recovery, and revision guidance, and the forward-only ablation already lifts Pass@1 from 50.0 to 56.7 on the 120-issue subset, so gains can arise without backward independence. The Limitations section acknowledges the same-backbone issue, but no experiment tests whether withholding I changes outcomes. If bI merely restates the generator's interpretation, RETRACE reduces to self-review with a structured revision plan, and the 'independent verification signal' claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces RETRACE, a training-free post-generation verification framework for coding-agent patches. After a patch p is generated from an issue I and trajectory τ, RETRACE (i) forward-reconstructs a repair rationale φ=(E,r,s) from I and the trajectory; (ii) backward-reconstructs the addressed problem bI = B(p,τ_b) with the original issue withheld; and (iii) computes an alignment verdict A(I,bI) and a reconciliation decision R that selects Submit or a targeted revision plan. Experiments on SWE-bench Verified report Pass@1 gains of 7.0 and 3.6 percentage points for GPT-5-mini and MiniMax M2.5 on mini-SWE-agent, additional gains on OpenHands, ablations on a random 120-issue subset, and a cost analysis. The paper claims that the improvement comes from the independent verification signal rather than from additional inference alone.","tokens_in":11821,"tokens_out":5294,"duration_ms":48603,"significance":"If the main claim holds, RETRACE would be a practical, scaffold-agnostic verification signal that does not require test execution or ground-truth patches; the cost analysis and ablations suggest the mechanism is inexpensive and that both stages matter. The design is mostly clean and uses an external standard benchmark, and the ablations on the 120-issue subset are a useful first decomposition of the gains. However, the load-bearing premise—that the backward reconstruction is truly independent of the original issue—is asserted rather than demonstrated, and the Self-Refine comparison does not isolate the claimed mechanism. No code or artifact is provided, so prompt-level reproducibility cannot be checked from the manuscript alone.","major_comments":[{"comment":"The central claim of an 'independent verification signal' is not supported by the experiments. The backward view τ_b is defined to exclude I and its explicit restatements, but τ_b still contains observations and decisions made while the agent had I available, and p is causally downstream of I. With the same backbone model performing B, bI can be a rationalization of the generator's own patch rather than an external check. The Limitations section acknowledges that both stages rely on the same backbone model and that the signal depends on issue quality, but no experiment varies whether I is withheld or uses a different verifier model. A control that compares RETRACE with and without access to I in the backward stage (or that uses a second model for bI and the alignment verdict) is needed before the Pass@1 gains can be attributed to independence rather than to extra structured inference.","section":"Backward Reconstruction of the Addressed Problem; Limitations"},{"comment":"The claim that the improvement stems from the verification signal rather than from additional computation alone is not established by the Self-Refine comparison. Self-Refine differs from RETRACE in prompt structure, evidence recovery, and revision guidance, and it is not tuned on mini-SWE-agent, where it degrades both baselines, whereas it does improve OpenHands. This confound prevents attributing the difference to the presence of a verification signal. A matched control using the same RETRACE prompts and revision loop but with access to I in the backward stage would isolate the withheld-issue mechanism.","section":"Experiments, Table 1 and Table 3; 'Performance of RETRACE'"},{"comment":"The quantitative claims rest on a single run per configuration with no confidence intervals, and the 120-issue ablation subset is described only as 'random' with no seed or selection procedure reported. With n=500, the headline 7.0-point gain has an approximate standard error of 2.2 points, but the OpenHands comparison and the ablations use n=120 with substantially larger uncertainty, and a single run cannot rule out seed- or scaffold-specific artifacts. Reporting multiple seeds or at least bootstrap confidence intervals, and fixing the subset seed in the experimental protocol, is needed to support the reported Pass@1 differences.","section":"Experimental Settings; Tables 2 and 3"}],"minor_comments":[{"comment":"The sentence 'evolutionary refinement (Zeng et al. 2025a), trajectory replay (Ding and Zhang 2025), distillation-conditioned rollouts (Gandhi et al. 2025a), or .' ends with an incomplete relative clause and an orphaned period.","section":"Related Work"},{"comment":"In the 'Evidence recovery' paragraph, the phrase 'retains the retains the code and program relationships' contains a duplicated fragment that should be corrected.","section":"Forward Reconstruction of Repair Reasoning"},{"comment":"The sentence 'despite using a comparable inference despite a comparable inference budget' is grammatically broken and should be rewritten.","section":"Performance of RETRACE"},{"comment":"Several references are duplicated with different labels (e.g., Gandhi et al. 2025a/2025b, CodeMonkeys 2025a/2025b, Trae Agent 2025a/2025b, Satori-SWE 2025a/2025b), and the author list for Li et al. 2026b contains the malformed entry 'Tse-Hsun; and Chen'; these should be consolidated and corrected.","section":"References"},{"comment":"The text refers to 'Figure 2' for the overlap analysis, but no Figure 2 image appears in the manuscript; the figure should be included or the reference removed.","section":"Ablation Studies"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism hinges on the independence of the backward reconstruction, which the current experiments do not demonstrate. I would look for a convincing withheld-issue control or cross-model verifier experiment before considering this for acceptance. The paper also contains several citation and formatting issues that should be cleaned in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on RETRACE. The core idea is genuinely new: after a coding agent produces a patch, RETRACE does a backward pass that reconstructs the problem statement from the patch and trajectory while withholding the original issue, then compares that reconstruction to the original issue to decide whether to submit or revise. That mechanism is not in the cited self-refinement or test-time scaling work, and it is a clever way to get a verification signal without tests or ground truth. The paper is also well-put-together: clean three-stage design, ablations showing both forward and backward stages contribute, a cost analysis showing no overhead, and transfer to a second scaffold without modification. Credit where due.\n\nThe soft spot is the load-bearing word “independent.” The backward reconstruction is done by the same backbone model that generated the patch, and the trajectory it uses was produced while the agent had the issue in context. Stripping “explicit restatements” from τ_b does not strip the model's own interpretation, because p and τ_b are causal outputs of a process that started from I. So the alignment score between bI and I can be a self-consistency check, not an independent verification. The paper's Limitations section acknowledges the same-backbone point, but no experiment tests the independence directly. The forward-only ablation already lifts Pass@1 by 6.7 points on the 120-issue subset, so the gains do not require the backward mechanism to be independent. And the Self-Refine baseline is not tuned on mini-SWE-agent — it degrades both backbones — so the claim that gains come from the verification signal rather than extra inference is weaker than it looks.\n\nThere are also experimental robustness issues: single run with greedy decoding, no confidence intervals, and a 120-issue ablation subset with no reported seed. These are fixable.\n\nOverall, the paper deserves a serious referee. The idea is useful and the direction is promising, but the current evidence does not establish the “independent verification signal” claim. I would condition acceptance on multiple seeds, code release, a better-tuned Self-Refine baseline, and either a heterogeneous verifier or an ablation that varies the model used for backward reconstruction. If the mechanism is reframed as structured self-verification, the empirical gains may hold, but the independence claim has to be earned.\n\nFor your reading group: worth a maybe. I would cite it if writing on coding agent verification.","headline":"A genuinely new backward-reconstruction verification idea with a load-bearing 'independence' claim the experiments don't yet support; deserves serious review, not a desk reject.","tokens_in":12335,"tokens_out":3013,"would_cite":true,"duration_ms":27314,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A coding agent can independently verify its own patch by reconstructing the problem the patch solves—with the original issue withheld—and comparing that reconstruction with the reported problem; on SWE-bench Verified this signal raises…","keywords":["RETRACE","coding agents","patch verification","bidirectional reconstruction","SWE-bench Verified","self-refinement","inference-time verification","program repair"],"falsifier":"Run RETRACE on a held-out set of SWE-bench issues and compare the fail-to-pass test pass rate of patches the framework labels 'Same' against those it labels 'Different'; if the 'Same' patches are not noticeably more likely to pass, the alignment verdict is not a correctness signal.","tokens_in":11398,"feed_emoji":"🐛","tokens_out":8664,"duration_ms":66913,"temperature":0.7,"pith_summary":"RETRACE is a training-free verification framework that gives a coding agent an independent check on its own patch: after the agent produces a patch, the framework reconstructs the repair rationale from the issue and trajectory, then reconstructs the problem the patch appears to address from the patch and trajectory alone (without the original issue), and reconciles the two views into a submit-or-revise decision. The central claim is that this bidirectional reconstruction produces a verification signal that predicts patch correctness without test execution or ground-truth patches. On SWE-bench Verified, the framework raises Pass@1 from 56.2% to 63.2% with GPT-5-mini and from 75.8% to 79.4% with MiniMax-2.5 on the mini-SWE-agent scaffold and delivers comparable gains on OpenHands. Ablations show both directions contribute, and the Self-Refine comparison is offered as evidence that the improvement comes from the verification signal, not from extra inference alone.","feed_headline":"Rebuilding the bug from a patch lifts coding-agent fixes by 7 points","feed_subtitle":"The agent that wrote the fix reconstructs the reported bug from the patch alone and compares—no tests, no ground truth.","key_machinery":"The mechanism that carries the argument is bidirectional reconstruction with reconciliation. Forward reconstruction takes the original issue $I$ and the forward trajectory view $\\tau_f$ and produces a repair rationale $\\phi=F(I,\\tau_f)=(E,r,s)$ consisting of recovered repository evidence, a logical reasoning chain, and an intended repair; backward reconstruction takes the candidate patch $p$ and the backward trajectory view $\\tau_b$ (which excludes the original issue) and produces $\\hat{I}=B(p,\\tau_b)$, a description of the problem the patch actually implements; reconciliation then computes the alignment verdict $v\\in\\{\\text{Same},\\text{Partial},\\text{Different}\\}$ between $\\hat{I}$ and $I$, checks the patch against $\\phi$, and selects a reconciliation action $a\\in\\{\\text{Submit},\\text{Revise-Patch},\\text{Revisit-Reasoning},\\text{Revise-Both}\\}$ along with an ordered revision plan. What this machinery does is convert the unanswerable question 'is this patch correct?' into a comparative question about whether the problem the patch solves matches the problem that was reported.","core_discovery":"On its own terms, the paper's discovery is that a patch's correctness can be read off the correspondence between two reconstructions of the same repair: a forward reconstruction $\\phi=F(I,\\tau_f)$ that recovers, from the original issue and the trajectory view, the evidence, reasoning chain, and intended repair, and a backward reconstruction $\\hat{I}=B(p,\\tau_b)$ that recovers, from the candidate patch and the backward trajectory view while withholding the original issue, the problem the patch appears to address. The reconciliation stage compares these and the original issue to emit a verdict (Same, Partial, Different) and a decision (Submit, Revise-Patch, Revisit-Reasoning, Revise-Both) with an ordered revision plan. The paper reports that on SWE-bench Verified this mechanism raises Pass@1 from 56.2% to 63.2% with GPT-5-mini and from 75.8% to 79.4% with MiniMax-2.5 on mini-SWE-agent, and from 37.5% to 56.7% and 62.5% to 70.0% on OpenHands with the two backbones.","pith_inferences":["A natural stress test is to decouple the verifier from the generator: run the backward reconstruction with a different backbone model than the one that produced the patch. If the gains persist, independence is real; if they vanish, the framework is largely restating the generator's own interpretation.","The same reconstruct-and-compare pattern could transfer to other generative tasks where the output must satisfy a hidden specification, such as summarization faithfulness, code review, or mathematical proof checking: ask the output to re-derive the question it answers, then compare. The paper does not claim this; it is an implication of the mechanism.","The alignment verdict could be calibrated against SWE-bench's fail-to-pass tests on a development subset to learn a confidence threshold for automatic submission, converting RETRACE from a revision loop into a selective verification oracle. This extends the paper's design rather than a claim it makes."],"forward_implications":["Coding agents can validate candidate patches without test execution or ground-truth patches, so verification works in settings where no reliable test oracle exists.","The framework is scaffold-agnostic: RETRACE improves Pass@1 on both mini-SWE-agent and OpenHands while leaving the underlying agent's tools and edit loop untouched.","Forward and backward reconstruction address complementary failure modes, and reconciliation rescues issues that neither direction resolves alone; removing either stage on the 120-issue subset lowers Pass@1 from 60.8% to 56.7%.","The gain is not explained by extra computation alone: with a comparable inference budget, Self-Refine fails to improve (and slightly lowers) the mini-SWE-agent baselines.","Because the added stages share cached inputs and reduce the agent's exploratory output, per-issue cost stays at or below the baseline despite the additional inference."],"supporting_citations":[{"why":"supplies the SWE-bench Verified benchmark with fail-to-pass and pass-to-pass tests used for evaluation.","marker":"(Jimenez et al. 2024)"},{"why":"supplies the SWE-agent/mini-SWE-agent scaffold with the bash-based agent-computer interface that RETRACE wraps.","marker":"(Yang et al. 2024)"},{"why":"is the self-refinement baseline used to show that the improvement comes from the verification signal rather than extra inference alone.","marker":"(Madaan et al. 2023)"},{"why":"supplies OpenHands, the second scaffold used to show scaffold-agnostic gains.","marker":"(Wang et al. 2025)"},{"why":"supplies the Pass@1 evaluation convention adopted in the experiments.","marker":"(Xia et al. 2025)"}],"fun_headline_variants":["Verify code patches by reconstructing the bug from patch alone","Patch fixes proven by re-deriving the original issue","Reconstruct the bug, verify the patch: coding agents get 7pt boost","Bidirectional reconstruction raises coding-agent Pass@1 by 7%","No tests needed: patch verified by rebuilding the bug report"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the backward reconstruction is genuinely independent: the same model that wrote the patch, when the original issue is hidden, can describe the problem the patch solves from the patch itself, and that description's match to the original issue predicts whether the patch is correct.","fun_headline_variants_meta":{"raw":{"variants":["Verify code patches by reconstructing the bug from patch alone","Patch fixes proven by re-deriving the original issue","Reconstruct the bug, verify the patch: coding agents get 7pt boost","Bidirectional reconstruction raises coding-agent Pass@1 by 7%","No tests needed: patch verified by rebuilding the bug report"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001042,"raw_usage":{"total_tokens":4456,"prompt_tokens":1091,"completion_tokens":3365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":3278}},"tokens_in":707,"tokens_out":3365,"duration_ms":23540,"temperature":1.0,"reasoning_tokens":3278,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:18:47.312137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RETRACE on a held-out set of SWE-bench issues and compare the fail-to-pass test pass rate of patches the framework labels 'Same' against those it labels 'Different'; if the 'Same' patches are not noticeably more likely to pass, the alignment verdict is not a correctness signal.","supporting_citations":[{"cited_title":"F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y.; Li, B.; Singh, J.; Tran, H","cited_arxiv_id":null,"evidence_quote":"supplies OpenHands, the second scaffold used to show scaffold-agnostic gains."}],"review_version":1}