{"id":"80c59828-9ac5-45f7-b28a-7c66207d00dd","arxiv_id":"2607.27083","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Cost-aware stopping over ranked tool prefixes, trained on stop-versus-continue payoff gaps, beats score and sufficiency thresholds under heterogeneous tool costs.","lead":"LLM agent harnesses can cut tool costs by learning where to stop on a ranked tool list, not just how to rank tools. CAM-DF turns existing rankings into cost-aware acquisition depths and, on retail agent tasks, cuts exposed tools by about 37% without clear success loss.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The central SOTA/payoff claim is load-bearing on Gx-exact sufficiency as both training target and metric; live n=67 only partially validates the proxy and does not establish non-inferior success.","rationale":"The reader correctly isolates the Gx/exact-sufficiency dependence as the weakest assumption behind the strongest claim. Theory (Thm 1 pathwise regret, Thm 2 Bayes alignment to E[Δ|X]) is internally clean given that target; Propositions 1–3 usefully show why score-only rules fail under heterogeneous costs. Experiments are broader than typical agent-tooling papers (five scorers, dense λ\times d grid, ablations, MCP-Atlas/WorkBench). None of that removes the circularity risk: offline SOTA is measured in the same currency used to build labels, and live evidence is supportive but not decisive (small n, no non-inferiority, residual success under missing tools). Author costs and occasional score-per-cost wins are real caveats but secondary—if Gx were a tight success proxy with powered live non-inferiority, the conditional verdict could move toward accept; if noise or live stratification shows the edge is mostly proxy artifact, it should move toward reject or a narrower claim (exposure reduction under a sufficiency proxy). I therefore leave the verdict CONDITIONAL and mark full agreement with the reader’s load-bearing concern. The concrete test uses artifacts the paper already ships and would settle the issue without new theory.","tokens_in":35401,"tokens_out":898,"duration_ms":17010,"concrete_test":"On the released Retail live logs (λ=0.12,d=1.0), recompute success stratified by exact Gx coverage for CAM-DF/CAM-DF-lite vs full/oracle, and run a TOST non-inferiority test of policy success vs full access with margin δ=0.05 (or bootstrap the success difference CI). Separately, inject the paper’s own membership noise (Table A19, ε∈{0.05,0.1}) into Gx, retrain, and re-score both offline payoff and the same live acquired sets; if live success drops >5pp or offline gains vs predict-then-threshold reverse while exposure stays low, the proxy-dependent claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim (highest deployable payoff on Retail; SOTA under heterogeneous costs/high λ; 37% fewer tools with comparable live success) rests on offline labels and metrics built from annotated minimal required sets Gx and U_exact=1{Gx⊆A}−λ∑cj (Sec. 3, Eq. 2; Algorithms A1–A2). Theorems 1–2 align the regret-weighted classifier to this clairvoyant gap, so if Gx is incomplete, over-specified, or only weakly predictive of realized success, both the stop/continue targets and the reported payoffs misstate the true decision problem. Live Retail (Tables A5–A6) is the main external check: oracle A=Gx yields 0.67 success vs 0.24 when one required tool is dropped (p<0.001), and CAM-DF reaches 0.67/0.63 under two simulators while exposing ~4.4 of 7 tools. But n=67 is underpowered for non-inferiority (McNemar vs full p=0.21/0.73), 4 CAM-DF tasks miss an annotated required tool yet still succeed, and insufficient still succeeds 21–24% of the time. Cross-domain labels are weaker still (trajectory consensus on WorkBench; dual-model shadow labels on Telecom). Author-chosen base costs and mixed cells vs tuned score-per-cost (Table 2, A18, A22–A23) are secondary; the Gx proxy is the single condition that, if soft, undercuts both the Bayes-aligned training story and the headline payoff/SOTA numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper formulates pre-execution tool acquisition for LLM agents as cost-aware stopping over ranked tool prefixes. Given an upstream ranking and heterogeneous tool costs, CAM-DF learns a stop/continue classifier from the offline payoff gap Δ(St)=U(At,x)−maxk>t U(Ak,x), using the sign as the label and |Δ| as a regret weight; CAM-DF-lite is a ten-feature interpretable variant. Theorems 1–2 give a pathwise prefix-regret decomposition and Bayes alignment of the weighted risk with sign(E[Δ|Xt]); Propositions 1–3 characterize the marginal value-per-cost rule and show score-monotone acquisition can be inconsistent under heterogeneous costs. Empirically, on τ-bench Retail and four additional domains (1,343 tasks), CAM-DF attains the highest mean payoff among deployable methods in the main Retail setting, with consistent gains over predict-then-threshold across ranking sources and cost regimes, and live execution reduces read-tool exposure by ~37% at comparable observed success.","tokens_in":35796,"tokens_out":1655,"duration_ms":41140,"significance":"If the results hold under the stated proxy, this is a useful modular control layer: it converts existing rankings into cost-aware acquisition depth without fine-tuning the LLM or changing the harness interface. The theory is standard cost-sensitive / decision-focused learning applied carefully rather than novel machinery, but the pathwise regret identity, Bayes alignment, and score-monotone failure propositions cleanly motivate the method and features. Strengths include extensive paired-bootstrap evaluation, multi-scorer robustness, ablations, cross-domain checks, live Retail execution with an independent simulator, full algorithmic appendices (A1–A2), and an interpretable lite variant with reported coefficients. The contribution is practical systems/ML for agent harnesses rather than a foundational theoretical breakthrough, but the problem framing (scores ≠ decisions under heterogeneous costs) is timely and well executed.","major_comments":[{"comment":"Sec. 3, Eq. (2), and Algorithms A1–A2: both training targets and the primary offline payoff rest on annotated minimal required sets Gx under exact sufficiency U=1{Gx⊆A}−λ∑cj. Theorems 1–2 align the learner to this clairvoyant gap, so the headline payoff/SOTA numbers are only as meaningful as Gx as a proxy for real task value. Live Retail (Tables A5–A6) shows a strong directional signal (oracle 0.67 vs insufficient 0.24, p<0.001) and CAM-DF success 0.63–0.67, but n=67 is underpowered for non-inferiority (McNemar vs full p=0.21/0.73), some CAM-DF sets miss annotated required tools yet succeed, and insufficient still succeeds ~21–24%. Cross-domain labels are weaker (WorkBench trajectory consensus; Telecom dual-model shadow labels). Please (i) state explicitly in the abstract and Sec. 6 that offline SOTA is under the Gx-sufficiency proxy, (ii) quantify proxy–success disagreement more systema","section":"Section 3, Eq. (2); Tables A5–A6, A19"},{"comment":"Abstract and Sec. 6.1 claim CAM-DF is “state-of-the-art under heterogeneous costs and high cost pressure” and attains the “highest payoff among deployable methods” on Retail. Table 2 and the dense grid (Fig. 3, Tables A18, A24) support consistent gains over predict-then-threshold and fixed-k, but the tuned score-per-cost threshold remains competitive at λ=0.12 (CIs often cover zero) and can lead or tie in parts of the grid and on Airline/MCP-Atlas partial-payoff cells (Tables A8, A22–A23). Please qualify the SOTA language to the baselines and regimes where paired CIs exclude zero, and avoid implying uniform dominance over all simple cost-aware heuristics.","section":"Abstract; Section 6.1; Table 2; Figure 3; Tables A18, A22–A24"},{"comment":"Live evaluation language (“maintaining comparable task success,” 37% fewer tools): Tables A5–A6 support reduced exposure (4.4 vs 7) with point estimates near full/oracle, but the paper correctly notes non-inferiority is not established. Soften abstract/conclusion wording to “no detectable drop in success in an underpowered n=67 check” (or similar), and flag that write/action tools remain unrestricted so the experiment isolates read-tool exposure only. One scorer/cost setting and ranked-prefix (not output-adaptive) scope should be stated next to the 37% figure.","section":"Abstract; Section 6.3; Tables A5–A6; Conclusion"}],"minor_comments":[{"comment":"Retail base costs (Table A2) are author-specified sensitivity weights, not measured telemetry. This is disclosed in the appendix; a one-sentence main-text note near the cost-regime discussion would help readers interpret heterogeneous-cost gains (the permutation check in App. D is reassuring and could be cited briefly).","section":"Section 6; Table A2; Appendix D"},{"comment":"Pipeline reconciliation (Table A4) shows nominally identical baselines differing slightly across released pipelines (e.g., predict-then-threshold 0.352 vs 0.361). Main-text tables should name the pipeline or state that all inference is within-pipeline paired, to avoid confusion when comparing across appendix tables.","section":"Table A4; Section 6"},{"comment":"Notation: U(A,x), v(A,Gx), and the exact/partial variants are clear, but Qstop/Qcontinue vs Ft and Δ(St) are introduced in close succession (Sec. 3–4); a small notation table or tighter cross-reference would help.","section":"Sections 3–4"},{"comment":"Figure 2 is effective; ensure the caption states λ=0.12, d=0 and that markers are mean tools vs mean payoff over tasks so readers do not read it as a causal frontier.","section":"Figure 2"},{"comment":"Related Work Table A1 is helpful; consider promoting a shortened version to the main text so positioning vs Stop-RAG / Adaptive-RAG / ranked-list truncation is visible without the supplement.","section":"Section 2; Table A1"},{"comment":"Typos/style: occasional missing spaces after commas in compound phrases in the abstract/intro (“agentharnessesface”, “rankingalonedoesnot”) appear to be PDF line-break artifacts; clean in the camera-ready.","section":"Abstract; Introduction"}],"recommendation":"minor_revision","confidential_remarks":"The Gx-proxy dependence is the real load-bearing risk, but the authors are more careful in the appendix than in the abstract; minor_revision with tightened claim language is proportionate. The work is a good fit for an ML systems / agent-tooling venue; novelty is in the problem formulation and deployable plugin rather than new learning theory. No integrity concerns; reproducibility artifacts appear unusually thorough for this area."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit here is simple: rankings are not acquisition decisions, and under heterogeneous tool costs you need a stop rule that looks at marginal value versus cost, not just score thresholds. CAM-DF turns an existing ranked list into a pre-execution prefix depth by training on the offline stop-versus-continue payoff gap, weighted by how much that mistake costs. That framing is the real contribution. The theory is mostly known cost-sensitive learning and marginal-value reasoning, but they apply it cleanly—pathwise regret decomposition, Bayes alignment of the |Δ|-weighted classifier, and a clear proof that score-monotone rules can invert the optimum when costs differ. I buy that package.\n\nEmpirically they did the work. Five domains, multiple frozen scorers, ablations on features/weights/objective, dense λ×d grids, and a live Retail run that cuts read-tool exposure ~37% with success rates that look comparable to full access. CAM-DF-lite is a nice practical touch. Gains over predict-then-threshold are consistent; gains over tuned score-per-cost are real mainly under high cost pressure and heterogeneity, which matches their theory. They do not oversell the homogeneous-cost cells.\n\nThe soft spot the stress-test flags is real and load-bearing, but not hidden. Offline labels and the main payoff both rest on annotated minimal required sets Gx and exact sufficiency. Live Retail partially validates that (oracle 0.67 vs drop-one 0.24), yet n=67 cannot establish non-inferiority, some tasks succeed without full Gx, and cross-domain labels are weaker (trajectory consensus, dual-model shadows). Author-chosen base costs are a secondary issue; they permute and still beat the learned baseline. If Gx is noisy in production, both the training target and the headline payoffs drift. That is a deployment caveat, not circular math—deployment features exclude Gx, and the alignment theorem is ordinary cost-sensitive Bayes, not a tautology.\n\nWho cares: people building agent harnesses, tool routers, and cost-aware RAG/tool stacks. Not a foundational ML result. Citations look appropriate; related work is honest about Adaptive-RAG, Stop-RAG, ranked-list truncation, and Pandora-style search.\n\nI would send this to peer review. It is a solid methods paper with reproducible structure and clear limits. Engage if you work on tool-using agents; skim the theory + Retail live section if you only need the control idea.","headline":"Clean, usable control layer for cost-aware tool depth; theory is standard but well applied, and the Gx proxy is the real soft spot—not a fake SOTA.","tokens_in":36458,"tokens_out":616,"would_cite":true,"duration_ms":18968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A ranking of tools is not an acquisition decision: stop where continuing no longer pays for its cost.","keywords":["LLM agents","tool acquisition","cost-aware stopping","decision-focused learning","ranked list depth","heterogeneous costs","regret-weighted classification"],"falsifier":"If, under verified required-set labels and heterogeneous costs, a tuned score-per-cost or sufficiency-threshold rule matched or beat CAM-DF’s mean payoff with confidence intervals covering zero across the Retail cost grid and the live agent no longer retained success while cutting tool exposure, the central claim would fail.","tokens_in":36245,"feed_emoji":"🛠️","tokens_out":907,"duration_ms":22710,"temperature":0.7,"pith_summary":"LLM agents face a practical control problem before any tool runs: how many of the ranked candidates should the harness actually expose? Too few leaves the task under-informed; too many burns cost, context, and privacy. This paper argues that relevance scores alone cannot settle that depth once tool costs differ, and that the right object is a stop-versus-continue decision along the ranked prefix. It trains a lightweight classifier on the offline payoff gap between stopping now and the best longer prefix, weighting each mistake by how much payoff is at stake, and proves that this target is Bayes-aligned with optimal stopping. On 1,343 tasks across five domains, the method leads deployable baselines under heterogeneous costs and high cost pressure, and in live Retail runs it cuts read-tool exposure by about 37% while keeping task success comparable. The point for a reader is modular: existing rankers stay frozen, the LLM stays frozen, and a small pre-execution plugin turns scores into cost-aware depth.","feed_headline":"Stop tool acquisition where the next call no longer pays","feed_subtitle":"A small pre-execution plugin turns rankings into cost-aware depth and cuts tool exposure ~37%","key_machinery":"CAM-DF (cost-aware marginal decision-focused stopping): a regret-weighted logistic stopper over ranked prefixes whose labels and weights come from Δ = stop payoff − best-continuation payoff; Theorem 1 decomposes path regret into weighted mistakes, and Theorem 2 shows the Bayes rule tracks sign(E[Δ|features]).","core_discovery":"Tool acquisition depth after ranking is a cost-aware stopping problem, not a score-threshold problem. Training on the signed payoff gap between the current prefix and its best continuation—weighted by the gap’s magnitude—yields a Bayes-aligned stopping rule that outperforms score-only and predict-then-threshold policies when costs are heterogeneous, without changing the upstream ranker or the underlying model.","pith_inferences":["The same stop-versus-continue gap could later be recomputed after each tool return, turning the one-shot prefix plugin into multi-round adaptive acquisition the paper flags as open.","If production systems only have noisy trajectory-mined required sets, the method’s edge may shrink toward simple cost heuristics unless labels are audited.","Harnesses that already log tool costs and outcomes could treat λ as an operator dial for latency or budget SLOs rather than a fixed research parameter."],"forward_implications":["Agent harnesses can add a frozen pre-execution stop layer on top of any existing router or retriever without fine-tuning the LLM.","Under heterogeneous tool prices, score-monotone acquisition is structurally inconsistent with optimal cost-aware subsets, so cost features must enter the stop decision.","Gains should widen when rankings are weaker, costs more dispersed, or cost pressure higher—the regimes the theory flags.","A compact ten-feature variant (CAM-DF-lite) can retain most of the full policy’s gains when interpretability or catalog stability matters.","Live deployment can cut exposed read tools by roughly a third while holding observed task success near full-access levels."],"fun_headline_variants":["Scores rank tools; costs decide how many to take","Stop tool calls when the next one no longer pays","Payoff-gap training beats score thresholds for tool depth","Cost-aware stopping cuts tool exposure 37% at same success","Ranked prefixes need a stopping rule, not a score cutoff"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"Training labels and the main offline payoff rest on knowing each task’s minimal required tool set and treating exact coverage of that set as the value of success.","fun_headline_variants_meta":{"raw":{"variants":["Scores rank tools; costs decide how many to take","Stop tool calls when the next one no longer pays","Payoff-gap training beats score thresholds for tool depth","Cost-aware stopping cuts tool exposure 37% at same success","Ranked prefixes need a stopping rule, not a score cutoff"]},"model":"grok-4.5","effort":"low","cost_usd":0.002946,"raw_usage":{"total_tokens":1090,"prompt_tokens":844,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":29464000,"prompt_tokens_details":{"text_tokens":844,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":181,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":844,"tokens_out":65,"duration_ms":5042,"temperature":1.0,"reasoning_tokens":181,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:38:50.178068+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If, under verified required-set labels and heterogeneous costs, a tuned score-per-cost or sufficiency-threshold rule matched or beat CAM-DF’s mean payoff with confidence intervals covering zero across the Retail cost grid and the live agent no longer retained success while cutting tool exposure, the central claim would fail.","supporting_citations":[],"review_version":1}