{"id":"acbdd3b8-b048-4572-aec6-34a13f02abae","arxiv_id":"2607.19338","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"Budget-calibrated recovery routing with conformal risk control lets coding agents match always-escalate solve rates at about 35% of the cost.","lead":"A new routing method tells coding agents whether to retry cheaply or escalate to an expensive model after failures, and uses conformal risk control to tune cost without retraining. Across five coding benchmarks it claims better solve rates at much lower cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exchangeability of CRC calibration with deployment-time failures is the linchpin; cost-control and frontier gains may not survive task drift.","rationale":"The reader's weakest assumption — that the CRC calibration and held-out failures are exchangeable with deployment-time failures — is exactly the most load-bearing concern. The paper's unique value proposition is the CRC layer's 'marginal expected-cost control under exchangeability,' and the empirical frontier improvement is demonstrated only on held-out data from the same five benchmarks. If the deployment distribution drifts, the cost-control guarantee is void and the router's advantage is ungrounded; the central claim would reduce to an in-distribution artifact. No other concern is more fundamental. I considered whether the 'one frontier point' using 35% cost could be a cherry-pick or a multiple-comparisons issue, but that is secondary and would not invalidate the existence of a favorable point; the exchangeability issue is more structural. Since the manuscript is abstract-only, no details are available to check statistical methodology or code, so the UNVERDICTED verdict is appropriate. My concern does not move the verdict; it reinforces that the paper cannot be accepted or rejected without full evidence, and that a drift experiment or explicit limitation would be needed to support the deployment-oriented claims.","tokens_in":646,"tokens_out":3507,"duration_ms":42077,"concrete_test":"Using the released code, take the trained router and CRC layer and evaluate on an out-of-distribution benchmark or a deliberately shifted version of one of the five benchmarks (e.g., altered difficulty, different repository, or corrupted feedback). Measure (a) the empirical mean recovery cost against the target alpha after applying the CRC-selected penalty, and (b) the full solve-rate/cost frontier relative to always-escalate and cascade baselines. If the empirical cost exceeds the nominal guarantee or the frontier ceases to dominate baselines, the exchangeability assumption is the load-bearing condition and the claims must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a supervised recovery router with a Conformal Risk Control (CRC) layer yields a calibrated cost/solve-rate frontier that beats fixed actions, prompt-only routers, and binary cascades. The formal component of this claim is CRC's 'marginal expected-cost control under exchangeability.' That guarantee holds only if calibration data and deployment-time failures are exchangeable. The abstract gives no evidence about distribution shift: no cross-benchmark experiments, no task-difficulty perturbations, no description of how the single '35% cost' frontier point was selected. In practice, coding-agent deployment encounters different repositories, bug distributions, and feedback quality than the five training benchmarks, so exchangeability is plausible but unverified. If deployment drifts, the CRC-selected cost penalty may no longer bound expected cost at the nominal level, and the router's decisions may be miscalibrated, potentially making the frontier worse than always-escalate. This is not an internal inconsistency; it is an external-validity condition that is load-bearing for the paper's deployment story. The abstract's strongest empirical claim rests on one held-out point without confidence intervals, so we cannot distinguish a robust frontier improvement from favorable selection in a single evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper 'CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents' proposes a supervised router that decides, after a failed coding-agent attempt, whether to spend more cheap compute or escalate to a stronger model. A Conformal Risk Control (CRC) layer selects a deployment-time cost penalty without retraining and is claimed to provide marginal expected-cost control under exchangeability. The authors report held-out evaluations across five coding benchmarks, asserting that the calibrated frontier beats fixed actions, prompt-only routers, and a binary cascade; in the main GPT-5.4-nano/GPT-5.4 setting, one frontier point claims a higher solve rate than always-escalate while using 35% of its mean recovery cost. Code is made available.","tokens_in":975,"tokens_out":2710,"duration_ms":31912,"significance":"If the claims are substantiated, this is a practical contribution: it offers a principled way to tune cost/solve-rate trade-offs in coding agents after execution failures, with a formal risk-control layer. A reproducible code release is a strength. However, the paper as submitted is abstract-only; no methods, proofs, or experimental details are available. The formal guarantee is conditional on exchangeability, which is not tested, and the headline empirical claim lacks uncertainty quantification. The significance is therefore conditional on the full text providing the missing evidence.","major_comments":[{"comment":"The submission consists solely of an abstract; there is no full text, no methods section, no derivation of the CRC bound, and no experimental protocol. This is not a reviewable journal manuscript. The central claims about the calibrated frontier and the CRC guarantee cannot be evaluated. A full paper with methodology, proofs, and reproducibility details is required before any soundness assessment.","section":"Abstract (entire manuscript)"},{"comment":"The abstract states that the CRC layer 'provides marginal expected-cost control under exchangeability.' Exchangeability is a load-bearing condition, but the paper gives no argument or experimental evidence that calibration-time failures and deployment-time failures are exchangeable. Five held-out benchmarks do not establish this; different repositories, bug distributions, and feedback quality could break the guarantee. Without a drift analysis or a formal weakening of the assumption, the cost-control claim is fragile and potentially vacuous in practical deployment.","section":"Abstract (CRC guarantee)"},{"comment":"The headline result — that one CRC-calibrated frontier point exceeds always-escalate solve rate while using 35% of its mean recovery cost — is reported without confidence intervals, number of held-out failures, or the protocol for selecting that point. The cost penalty is selected via CRC on calibration data, so we need to know whether this point is a cherry-picked extremum. The broader claim that the calibrated frontier improves over fixed actions, prompt-only routers, and a binary cascade also lacks any uncertainty quantification across benchmarks and repetitions. These are necessary to distinguish a robust effect from chance.","section":"Abstract (empirical claim)"}],"minor_comments":[{"comment":"The five coding benchmarks are not named. In the full paper, identify them and specify which are used for training, calibration, and evaluation.","section":"Abstract (benchmarks)"},{"comment":"The GPT-5.4-nano/GPT-5.4 setting is unclear; clarify the model family, API costs, and why these two are the 'main' setting.","section":"Abstract (models)"},{"comment":"The GitHub link is useful, but for review the paper should include a reproducibility statement, data splits, and a specification of compute/API usage.","section":"Abstract (code)"}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only submission, which is insufficient for journal review. The exchangeability assumption is the core risk; the authors should be required to provide cross-benchmark or distribution-shift experiments and a clear statement of when the CRC guarantee applies. The empirical claim also needs error bars and a pre-registered or clearly specified selection protocol. I recommend asking for a full manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nI read the abstract for CodeRescue. The short version: this is a real contribution to the coding-agents cost literature, and I'd send it to referees rather than desk-reject.\n\nWhat's new: the paper frames post-failure decisions as recovery routing over heterogeneous actions — spend more cheap compute vs escalate — which is a step beyond the usual cascade. Adding CRC to select a cost penalty at deployment time without retraining is a neat trick; it makes the same router budget-adjustable. The abstract claims a calibrated frontier beats fixed actions, prompt-only routers, and a binary cascade, with a specific point at 35% mean recovery cost while exceeding escalate solve rate. That's the kind of concrete claim that gets attention.\n\nThe good parts: the abstract is honest about the setup — trained on execution rollouts, evaluated on held-out failures. That addresses the circularity worry up front. Code is linked. No obvious red flags.\n\nSoft spots: this is abstract-only, so I can't verify the empirical claim. No error bars, no description of how the single frontier point was selected, and the formal CRC guarantee rests on exchangeability between calibration and deployment. The stress-test note about distribution shift is the right thing to ask: if deployment tasks differ from the five benchmarks, the cost control can degrade. That's an external validity condition, not an internal inconsistency. The paper would be stronger with cross-benchmark or shifted-task experiments, but I wouldn't call it a fatal flaw.\n\nOne more thing: the router is trained on rollouts from the same benchmarks it's evaluated on. That's standard supervised learning, not circular, as long as the evaluation uses a held-out split — which the abstract says it does. I'd want the full details on that split.\n\nWho should read it? Anyone working on cost-aware LLM agents, model cascading, or budgeted inference. It's a practical paper, not a theoretical breakthrough. If the full text matches the abstract, it's a useful baseline for future work.\n\nMy recommendation: send it to peer review. Ask for error bars, the CRC implementation details, and ideally some distribution-shift analysis. But the idea is sound and the abstract is coherent enough to justify referee time.\n\nBest,\n[Your name]","headline":"Recovery routing with CRC budget calibration is a genuinely new idea; the abstract is coherent and deserves referee time, though the empirical claims need full-text verification.","tokens_in":1388,"tokens_out":2489,"would_cite":false,"duration_ms":25421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Post-failure routing with a conformal risk-control layer lets coding agents beat always-escalate on solve rate while spending about one-third of the recovery cost.","keywords":["recovery routing","conformal risk control","coding agents","execution feedback","budget calibration","model escalation","test-time compute allocation"],"falsifier":"Run the CRC-calibrated router on a new coding benchmark whose failure mix differs from the five used for calibration (e.g., a different language, much longer traces, or adversarially selected failures) and compare the realized mean recovery cost to the calibrated marginal bound. If the realized cost repeatedly exceeds the bound, the exchangeability assumption failed; a single run under the bound would not settle it.","tokens_in":1308,"feed_emoji":"🤖","tokens_out":2357,"duration_ms":57160,"temperature":0.7,"pith_summary":"This paper argues that when a coding agent fails, the next action should be a routing decision: spend more cheap compute on another attempt, or escalate to a stronger, more expensive model. It trains a supervised router from execution rollouts and adds a Conformal Risk Control (CRC) layer that tunes a deployment-time cost penalty without retraining, giving a marginal expected-cost guarantee under exchangeability. On held-out failures from five coding benchmarks, the calibrated cost/solve-rate frontier improves over fixed actions, prompt-only routers, and a binary cascading baseline. In the main GPT-5.4-nano/GPT-5.4 setting, one calibrated point exceeds the always-escalate solve rate while using only 35% of its mean recovery cost. This matters because it replaces the usual escalation ladder with a budget-adaptive policy that treats execution feedback as a resource worth spending on.","feed_headline":"Calibrated router beats escalation at 35% of cost","feed_subtitle":"It keeps solve rate above always-escalate while spending a third of the recovery cost.","key_machinery":"The central object is a recovery router over heterogeneous actions — cheap-model retry vs escalation to a stronger model — supervised on execution rollouts, together with a Conformal Risk Control (CRC) layer. The CRC layer selects a deployment-time cost penalty, converting the pretrained router into a spectrum of operating points on the cost/solve-rate frontier and attaching a formal marginal expected-cost guarantee under exchangeability. Its work is to make budget changes a choice of a penalty parameter rather than a retraining event, while preserving a statistical control on mean recovery cost.","core_discovery":"The paper establishes that a single supervised router over recovery actions (cheap retry versus escalation), combined with a Conformal Risk Control layer that selects a deployment cost penalty, yields a full frontier of cost/solve-rate trade-offs for post-failure decisions in coding agents. The calibrated frontier dominates fixed actions, prompt-only routers, and a binary cascade baseline across held-out failures from five coding benchmarks. In the main GPT-5.4-nano/GPT-5.4 setting, a CRC-calibrated point achieves a higher solve rate than always-escalate while spending only 35% of its mean recovery cost. Under exchangeability of held-out and deployment failures, the CRC layer provides margin","pith_inferences":["The same routing-plus-CRC design could generalize to other agent settings with execution feedback and heterogeneous action costs — theorem proving, tool-use agents, or robotics retries — wherever a failed attempt yields actionable signal.","An implicit policy claim worth testing: the escalation-ladder architecture is suboptimal because execution feedback carries information; a router that distinguishes retryable from escalate-only failures exploits that information, implying future systems should allocate compute based on feedback content, not just cost tiers.","The guarantee is marginal expected cost, not per-task worst case; a deployment team needing a per-task cost cap would require a different control scheme — a boundary the paper leaves implicit.","A testable extension: apply CRC to the router's confidence threshold for retry versus escalate instead of a cost penalty, and compare the resulting frontier shape and guarantee strength."],"forward_implications":["Cheap recovery and escalation find different failure modes, so the best deployment policy mixes them according to budget rather than following a strict escalation ladder.","The CRC layer lets a single trained router adapt to new budgets at deployment time without retraining, with a formal bound on mean recovery cost under exchangeability.","Under exchangeability, deployment teams get a statistical cost guarantee alongside a solve-rate target, making budget planning risk-aware.","The main GPT-5.4-nano/GPT-5.4 result indicates that always-escalate is not optimal when cost matters; a calibrated mix achieves higher solve rate at one-third of the recovery cost.","A router trained on execution rollouts transfers across five coding benchmarks, suggesting that recovery-routing policies generalize across coding task distributions."],"fun_headline_variants":["Router with risk control beats escalation at third the cost","Conformal router: higher solve rate, 65% cheaper recovery","One router, any budget: calibration beats cascade on code fixes","Cost-calibrated router outperforms escalation on coding failures","Cheap recovery routed smartly: matching escalate success for 35% cost"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The marginal expected-cost guarantee holds only if deployment-time failures are exchangeable with the held-out failures used for calibration; if the real task distribution drifts, the guarantee and the calibrated frontier transfer break.","fun_headline_variants_meta":{"raw":{"variants":["Router with risk control beats escalation at third the cost","Conformal router: higher solve rate, 65% cheaper recovery","One router, any budget: calibration beats cascade on code fixes","Cost-calibrated router outperforms escalation on coding failures","Cheap recovery routed smartly: matching escalate success for 35% cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2411,"prompt_tokens":745,"completion_tokens":1666,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1579}},"tokens_in":489,"tokens_out":1666,"duration_ms":12023,"temperature":1.0,"reasoning_tokens":1579,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:13:12.925051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the CRC-calibrated router on a new coding benchmark whose failure mix differs from the five used for calibration (e.g., a different language, much longer traces, or adversarially selected failures) and compare the realized mean recovery cost to the calibrated marginal bound. If the realized cost repeatedly exceeds the bound, the exchangeability assumption failed; a single run under the bound would not settle it.","supporting_citations":[],"review_version":2}