CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

Chenqian Le; Jiayi Cheng; Jie Mei; Qijia He; Rui Wang; Tao Wang; XunMei Liu; Xupeng Chen; Yixian Chen; Yuhuan Chen

Post-failure routing with a conformal risk-control layer lets coding agents beat always-escalate on solve rate while spending about one-third of the recovery cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:13 UTC pith:YMGTSDBU

load-bearing objection Recovery routing with CRC budget calibration is a genuinely new idea; the abstract is coherent and deserves referee time, though the empirical claims need full-text verification. the 3 major comments →

arxiv 2607.19338 v2 pith:YMGTSDBU submitted 2026-07-21 cs.AI

CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents

Qijia He , Jiayi Cheng , Chenqian Le , Rui Wang , Xunmei Liu , Yixian Chen , Jie Mei , Zhihao Wang

show 3 more authors

Xupeng Chen Yuhuan Chen Tao Wang

This is my paper

classification cs.AI

keywords recovery routingconformal risk controlcoding agentsexecution feedbackbudget calibrationmodel escalationtest-time compute allocation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when a coding agent fails, the next action should be a routing decision: spend more cheap compute on another attempt, or escalate to a stronger, more expensive model. It trains a supervised router from execution rollouts and adds a Conformal Risk Control (CRC) layer that tunes a deployment-time cost penalty without retraining, giving a marginal expected-cost guarantee under exchangeability. On held-out failures from five coding benchmarks, the calibrated cost/solve-rate frontier improves over fixed actions, prompt-only routers, and a binary cascading baseline. In the main GPT-5.4-nano/GPT-5.4 setting, one calibrated point exceeds the always-escalate solve rate while using only 35% of its mean recovery cost. This matters because it replaces the usual escalation ladder with a budget-adaptive policy that treats execution feedback as a resource worth spending on.

Core claim

The paper establishes that a single supervised router over recovery actions (cheap retry versus escalation), combined with a Conformal Risk Control layer that selects a deployment cost penalty, yields a full frontier of cost/solve-rate trade-offs for post-failure decisions in coding agents. The calibrated frontier dominates fixed actions, prompt-only routers, and a binary cascade baseline across held-out failures from five coding benchmarks. In the main GPT-5.4-nano/GPT-5.4 setting, a CRC-calibrated point achieves a higher solve rate than always-escalate while spending only 35% of its mean recovery cost. Under exchangeability of held-out and deployment failures, the CRC layer provides margin

What carries the argument

The central object is a recovery router over heterogeneous actions — cheap-model retry vs escalation to a stronger model — supervised on execution rollouts, together with a Conformal Risk Control (CRC) layer. The CRC layer selects a deployment-time cost penalty, converting the pretrained router into a spectrum of operating points on the cost/solve-rate frontier and attaching a formal marginal expected-cost guarantee under exchangeability. Its work is to make budget changes a choice of a penalty parameter rather than a retraining event, while preserving a statistical control on mean recovery cost.

Load-bearing premise

The marginal expected-cost guarantee holds only if deployment-time failures are exchangeable with the held-out failures used for calibration; if the real task distribution drifts, the guarantee and the calibrated frontier transfer break.

What would settle it

Run the CRC-calibrated router on a new coding benchmark whose failure mix differs from the five used for calibration (e.g., a different language, much longer traces, or adversarially selected failures) and compare the realized mean recovery cost to the calibrated marginal bound. If the realized cost repeatedly exceeds the bound, the exchangeability assumption failed; a single run under the bound would not settle it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Cheap recovery and escalation find different failure modes, so the best deployment policy mixes them according to budget rather than following a strict escalation ladder.
The CRC layer lets a single trained router adapt to new budgets at deployment time without retraining, with a formal bound on mean recovery cost under exchangeability.
Under exchangeability, deployment teams get a statistical cost guarantee alongside a solve-rate target, making budget planning risk-aware.
The main GPT-5.4-nano/GPT-5.4 result indicates that always-escalate is not optimal when cost matters; a calibrated mix achieves higher solve rate at one-third of the recovery cost.
A router trained on execution rollouts transfers across five coding benchmarks, suggesting that recovery-routing policies generalize across coding task distributions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same routing-plus-CRC design could generalize to other agent settings with execution feedback and heterogeneous action costs — theorem proving, tool-use agents, or robotics retries — wherever a failed attempt yields actionable signal.
An implicit policy claim worth testing: the escalation-ladder architecture is suboptimal because execution feedback carries information; a router that distinguishes retryable from escalate-only failures exploits that information, implying future systems should allocate compute based on feedback content, not just cost tiers.
The guarantee is marginal expected cost, not per-task worst case; a deployment team needing a per-task cost cap would require a different control scheme — a boundary the paper leaves implicit.
A testable extension: apply CRC to the router's confidence threshold for retry versus escalate instead of a cost penalty, and compare the resulting frontier shape and guarantee strength.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Recovery routing with CRC budget calibration is a genuinely new idea; the abstract is coherent and deserves referee time, though the empirical claims need full-text verification.

read the letter

Colleague,

I read the abstract for CodeRescue. The short version: this is a real contribution to the coding-agents cost literature, and I'd send it to referees rather than desk-reject.

What's new: the paper frames post-failure decisions as recovery routing over heterogeneous actions — spend more cheap compute vs escalate — which is a step beyond the usual cascade. Adding CRC to select a cost penalty at deployment time without retraining is a neat trick; it makes the same router budget-adjustable. The abstract claims a calibrated frontier beats fixed actions, prompt-only routers, and a binary cascade, with a specific point at 35% mean recovery cost while exceeding escalate solve rate. That's the kind of concrete claim that gets attention.

The good parts: the abstract is honest about the setup — trained on execution rollouts, evaluated on held-out failures. That addresses the circularity worry up front. Code is linked. No obvious red flags.

Soft spots: this is abstract-only, so I can't verify the empirical claim. No error bars, no description of how the single frontier point was selected, and the formal CRC guarantee rests on exchangeability between calibration and deployment. The stress-test note about distribution shift is the right thing to ask: if deployment tasks differ from the five benchmarks, the cost control can degrade. That's an external validity condition, not an internal inconsistency. The paper would be stronger with cross-benchmark or shifted-task experiments, but I wouldn't call it a fatal flaw.

One more thing: the router is trained on rollouts from the same benchmarks it's evaluated on. That's standard supervised learning, not circular, as long as the evaluation uses a held-out split — which the abstract says it does. I'd want the full details on that split.

Who should read it? Anyone working on cost-aware LLM agents, model cascading, or budgeted inference. It's a practical paper, not a theoretical breakthrough. If the full text matches the abstract, it's a useful baseline for future work.

My recommendation: send it to peer review. Ask for error bars, the CRC implementation details, and ideally some distribution-shift analysis. But the idea is sound and the abstract is coherent enough to justify referee time.

Best, [Your name]

Referee Report

3 major / 3 minor

Summary. The paper 'CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents' proposes a supervised router that decides, after a failed coding-agent attempt, whether to spend more cheap compute or escalate to a stronger model. A Conformal Risk Control (CRC) layer selects a deployment-time cost penalty without retraining and is claimed to provide marginal expected-cost control under exchangeability. The authors report held-out evaluations across five coding benchmarks, asserting that the calibrated frontier beats fixed actions, prompt-only routers, and a binary cascade; in the main GPT-5.4-nano/GPT-5.4 setting, one frontier point claims a higher solve rate than always-escalate while using 35% of its mean recovery cost. Code is made available.

Significance. If the claims are substantiated, this is a practical contribution: it offers a principled way to tune cost/solve-rate trade-offs in coding agents after execution failures, with a formal risk-control layer. A reproducible code release is a strength. However, the paper as submitted is abstract-only; no methods, proofs, or experimental details are available. The formal guarantee is conditional on exchangeability, which is not tested, and the headline empirical claim lacks uncertainty quantification. The significance is therefore conditional on the full text providing the missing evidence.

major comments (3)

[Abstract (entire manuscript)] The submission consists solely of an abstract; there is no full text, no methods section, no derivation of the CRC bound, and no experimental protocol. This is not a reviewable journal manuscript. The central claims about the calibrated frontier and the CRC guarantee cannot be evaluated. A full paper with methodology, proofs, and reproducibility details is required before any soundness assessment.
[Abstract (CRC guarantee)] The abstract states that the CRC layer 'provides marginal expected-cost control under exchangeability.' Exchangeability is a load-bearing condition, but the paper gives no argument or experimental evidence that calibration-time failures and deployment-time failures are exchangeable. Five held-out benchmarks do not establish this; different repositories, bug distributions, and feedback quality could break the guarantee. Without a drift analysis or a formal weakening of the assumption, the cost-control claim is fragile and potentially vacuous in practical deployment.
[Abstract (empirical claim)] The headline result — that one CRC-calibrated frontier point exceeds always-escalate solve rate while using 35% of its mean recovery cost — is reported without confidence intervals, number of held-out failures, or the protocol for selecting that point. The cost penalty is selected via CRC on calibration data, so we need to know whether this point is a cherry-picked extremum. The broader claim that the calibrated frontier improves over fixed actions, prompt-only routers, and a binary cascade also lacks any uncertainty quantification across benchmarks and repetitions. These are necessary to distinguish a robust effect from chance.

minor comments (3)

[Abstract (benchmarks)] The five coding benchmarks are not named. In the full paper, identify them and specify which are used for training, calibration, and evaluation.
[Abstract (models)] The GPT-5.4-nano/GPT-5.4 setting is unclear; clarify the model family, API costs, and why these two are the 'main' setting.
[Abstract (code)] The GitHub link is useful, but for review the paper should include a reproducibility statement, data splits, and a specification of compute/API usage.

Circularity Check

0 steps flagged

No circularity identifiable from the abstract; the supervised router is evaluated on held-out failures and the CRC layer's guarantee rests on a stated exchangeability assumption, which is an external-validity condition rather than a reduction to inputs.

full rationale

This is an abstract-only review, so only the abstract's claims can be examined. The derivation chain described is: (1) train a supervised recovery router from execution rollouts, (2) add a Conformal Risk Control layer that selects a deployment-time cost penalty on calibration data, and (3) evaluate across held-out failures from five coding benchmarks. This is standard supervised evaluation plus conformal calibration: the router and penalty are fit on training/calibration data and assessed on held-out failures. The held-out step prevents the reported solve-rate and cost improvements from being forced by construction. The CRC guarantee is explicitly conditioned on exchangeability, which is a stated assumption about future data, not a circular definition of the target result. No equation is quoted that defines a predicted quantity in terms of the fitted parameter, and the abstract does not invoke any load-bearing self-citation or rename a known result. The main legitimate concern is external validity and possible selection of the headline 'one frontier point,' but that is a robustness/selection-bias issue, not circularity. Under the instruction to flag only circularity that can be exhibited by quote and specific reduction, no such step is present. Score 0.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claim is empirical: a supervised router plus conformal calibration. The paper introduces no new physical entities. It relies on standard statistical assumptions, most importantly exchangeability, and on externally specified cost models.

free parameters (2)

supervised router parameters = unknown
The router is trained from execution rollouts; the learned parameters determine the routing policy and are fitted to data.
cost penalty selected by CRC = unknown
A deployment-time cost penalty is chosen via Conformal Risk Control on calibration data to meet an expected-cost bound; the value is data-dependent.

axioms (4)

domain assumption Exchangeability of calibration rollouts and deployment-time failures
The abstract states that CRC 'provides marginal expected-cost control under exchangeability'; if this fails, the cost guarantee is void.
domain assumption Execution feedback contains signal for recovery decisions
The entire method assumes that failed attempts produce actionable feedback that makes further cheap-model recovery worthwhile in at least some states.
domain assumption Benchmarks are representative of deployment workloads
The reported success on five coding benchmarks is used to argue general effectiveness; no distribution shift analysis is visible in the abstract.
domain assumption Action costs are known and comparable
The '35% of mean recovery cost' statement presumes a defined cost model for cheap recovery versus escalation, which the user must supply.

pith-pipeline@v1.3.0-alltime-deepseek · 563 in / 6364 out tokens · 59821 ms · 2026-08-03T00:13:12.925051+00:00 · methodology

0 comments

read the original abstract

Coding agents increasingly operate in executable environments where a failed attempt produces actionable feedback rather than merely an incorrect answer. Existing cost-aware systems typically treat such failures as cascade decisions: try a cheap model first, then escalate hard cases to a stronger and more expensive model. In coding, however, execution feedback can also make further cheap-model recovery worthwhile, raising a budgeted deployment question: when should an agent spend more cheap compute, and when should it escalate? We formulate this post-failure decision as recovery routing over heterogeneous actions and train a supervised router from execution rollouts. To make the same router usable under changing budgets, we add a Conformal Risk Control (CRC) layer that selects a deployment-time cost penalty without retraining and provides marginal expected-cost control under exchangeability. Across held-out failures from five coding benchmarks, cheap recovery and escalation exhibit complementary success patterns. The calibrated frontier improves over fixed actions, prompt-only routers, and a binary cascade baseline; in the main GPT-5.4-nano/GPT-5.4 setting, one CRC-calibrated frontier point exceeds always-escalate solve rate while using 35% of its mean recovery cost. Code is available at https://github.com/Qijia-He/agent-budget-control.

Figures

Figures reproduced from arXiv: 2607.19338 by Chenqian Le, Jiayi Cheng, Jie Mei, Qijia He, Rui Wang, Tao Wang, XunMei Liu, Xupeng Chen, Yixian Chen, Yuhuan Chen, Zhihao Wang.