{"id":"f3f29e78-c412-4337-b630-73f3bfb75e55","arxiv_id":"2606.00198","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Frontier LLM agents lack budget-awareness, over-spend on failing trajectories, and can be improved with SFT+RL but still show only 47% interval coverage.","lead":"The paper defines budget-aware LLM agents that actively estimate remaining compute and action costs at each step and alert when success is unlikely. It tests frontier agents on four environments and finds they are over-optimistic, continue spending on failing tasks, and that training can improve early stopping but not precise interval calibration.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Rollout-replay protocol and four environments lack demonstrated external validity for real budget deployment","rationale":"The reader's weakest_assumption already isolates the precise external-validity gap that bears on the strongest_claim. No deeper internal inconsistency (e.g., circular definition of 'unlikely to succeed' or mis-specified interval scoring) is visible in the abstract and verdict summary, so the existing CONDITIONAL/LOW verdict stands.","tokens_in":1712,"tokens_out":365,"duration_ms":15669,"concrete_test":"Re-run the five agents on the same four environments but with an enforced live budget cap (stop execution and record alert time when cumulative cost exceeds a pre-set threshold); compare early-alert frequency and token savings against the original rollout-replay scores. A >20 % absolute difference in alert rate on failed trajectories would indicate the protocol does not generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim (frontier models are over-optimistic and fail to alert early) is measured exclusively via the rollout-replay protocol on four environments. This protocol scores progressive interval estimates and early-alert behavior on recorded trajectories. For the claim to hold beyond the testbed, two conditions must obtain: (1) replayed budget predictions must match the decisions an agent would make under live budget constraints, and (2) the chosen environments must induce budget dynamics representative of deployment. Neither is shown; no ablation compares replay scores to actual budget-capped runs, and no argument is given that the four environments span the relevant distribution of task lengths, failure modes, or cost structures. If either condition fails, the observed over-optimism and low interval coverage (47 % post-SFT+RL) may be artifacts of the test harness rather than intrinsic model properties.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper defines budget estimation (internal from computation, external from actions) and budget-awareness as progressive interval estimation with early alerts on unlikely completion. Using a rollout-replay protocol on four environments and five frontier agents, it reports r=0.35 correlation between agent capability and budget-awareness, consistent over-optimism (continued spending on failing tasks), 28-64% token savings from early stopping, and that SFT+RL improves alerting but caps interval coverage at 47%.","tokens_in":1904,"tokens_out":458,"duration_ms":9616,"significance":"If the measurements generalize, the work identifies a practically important limitation in current LLM agents and demonstrates that budget-awareness is trainable, which could inform more resource-efficient agent systems. The concrete numbers on savings and coverage provide a clear baseline for future work.","major_comments":[{"comment":"§3 (rollout-replay protocol): the central claim that frontier models are over-optimistic and fail to alert early rests entirely on replay scores from recorded trajectories; no ablation or comparison to live budget-capped executions is reported, so it is unclear whether the observed 47% coverage and over-optimism reflect intrinsic model behavior or artifacts of the replay harness.","section":"§3"},{"comment":"§4 (environments): the four chosen environments are used to support generalization claims about budget dynamics, yet no justification, ablation, or coverage argument is given that they span representative task lengths, failure modes, or cost structures; this directly affects whether the r=0.35 correlation and early-stop savings hold beyond the testbed.","section":"§4"}],"minor_comments":[{"comment":"The abstract states 'precise interval calibration remains challenging' but the methods section should explicitly define how interval bounds are elicited from the model at each step.","section":"Methods"},{"comment":"Table reporting per-agent/per-environment results should include raw counts of trajectories and failure rates to allow assessment of statistical power behind the 28-64% savings range.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our work. Below we respond point-by-point to the major comments, proposing revisions where the manuscript can be strengthened without misrepresenting our current results.","responses":[{"response":"The rollout-replay protocol was designed to isolate the agent's internal budget estimation and alerting behavior by replaying fixed trajectories, thereby removing confounding variables such as live API variability or external system state changes. This enables precise scoring of interval predictions against ground-truth remaining costs. We acknowledge that the lack of a direct comparison to live budget-capped executions leaves open the possibility of harness-specific artifacts, and this constitutes a genuine limitation of the current evaluation. We will revise §3 to articulate the rationale for replay, add an explicit limitations paragraph discussing potential differences from live settings, and outline future work on live evaluations.","revision_made":"partial","referee_comment":"[§3] §3 (rollout-replay protocol): the central claim that frontier models are over-optimistic and fail to alert early rests entirely on replay scores from recorded trajectories; no ablation or comparison to live budget-capped executions is reported, so it is unclear whether the observed 47% coverage and over-optimism reflect intrinsic model behavior or artifacts of the replay harness."},{"response":"We selected the four environments for their established use in agent benchmarks and their differing action and token cost profiles, but the manuscript indeed provides no explicit justification, ablation, or coverage analysis of task lengths and failure modes. We agree this weakens the generalization argument. We will add a dedicated paragraph to §4 that describes the task-length distributions, primary failure modes, and cost structures of each environment, thereby supporting the reported correlation and savings figures.","revision_made":"yes","referee_comment":"[§4] §4 (environments): the four chosen environments are used to support generalization claims about budget dynamics, yet no justification, ablation, or coverage argument is given that they span representative task lengths, failure modes, or cost structures; this directly affects whether the r=0.35 correlation and early-stop savings hold beyond the testbed."}],"tokens_in":1351,"tokens_out":458,"duration_ms":15727,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core point is that this work defines budget-awareness as progressive interval estimation of remaining cost at each step, then uses a rollout-replay protocol to score five frontier agents across four environments. It reports low correlation (r=0.35) between task performance and budget awareness, consistent over-optimism on failing trajectories, 28-64% token savings from early stopping, and a lift from SFT+RL that still leaves interval coverage at only 47%.\n\nWhat stands out as new is the interval-estimation framing itself and the replay scoring method; these are not standard extensions of existing agent benchmarks. The empirical numbers on savings and the effect of training are concrete and directly usable for anyone trying to cap costs in production agents.\n\nThe main limitation is external validity. The abstract gives no ablation showing that replay scores align with decisions made under live budget caps, and no argument that the four environments cover representative task lengths or cost structures. If those conditions do not hold, the over-optimism pattern and the 47% coverage figure may not travel. The low r value also indicates the signal is noisy even inside the testbed.\n\nThis paper is aimed at researchers building or evaluating LLM agents who need a practical handle on cost control. It is worth sending to peer review because the problem is real, the protocol is new, and the training results are falsifiable; a referee can check the methods and ask for the missing generalization tests.","headline":"The paper supplies a fresh protocol for measuring budget-awareness in agents and shows over-optimism plus trainable early stopping, but the rollout-replay results rest on four environments whose match to real deployments is untested.","tokens_in":2459,"tokens_out":381,"would_cite":false,"duration_ms":16704,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Frontier LLM agents are over-optimistic about budgets and rarely alert users before wasting resources on failing tasks.","keywords":["budget-aware agents","LLM agents","cost estimation","interval estimation","early stopping","over-optimism","frontier models","token efficiency"],"falsifier":"Live deployment of the same agents on real user tasks where predicted intervals and alerts are compared against the actual remaining budget at the moment each alert is issued.","tokens_in":2613,"feed_emoji":"💰","tokens_out":705,"duration_ms":14132,"temperature":0.7,"pith_summary":"The paper tests whether LLM agents can treat budgets as an active signal during planning rather than a number recorded after the fact. It formalizes budget-awareness as the ability to output shrinking interval estimates of remaining cost at each step and to flag when success becomes improbable. Experiments across five frontier models and four environments show that agent capability correlates only weakly with this skill, that models continue spending on doomed trajectories, and that the signal can be improved through training even if precise calibration stays difficult. A rollout-replay protocol supplies the measurements, and early-stop interventions demonstrate concrete token savings on failed runs.","feed_headline":"Frontier agents keep spending after failure is likely","feed_subtitle":"Tests on five models show weak budget awareness; early stopping recovers 28-64% of tokens on unsuccessful trajectories.","key_machinery":"Progressive interval estimation: at each step the agent predicts upper and lower bounds on remaining budget and alerts when success probability drops.","core_discovery":"Budget-awareness is defined as progressive interval estimation: at every planning step an agent must output an upper and lower bound on remaining budget and issue an alert once completion is unlikely. Five frontier agents evaluated on four environments under a rollout-replay protocol exhibit only modest correlation (r=0.35) between task performance and budget estimation quality; the models are systematically over-optimistic and keep consuming resources instead of alerting early. The budget signal proves actionable, with early stopping recovering 28-64% of tokens on failed trajectories, and SFT+RL training strengthens stopping and alerting behavior, although interval coverage reaches only 47%","pith_inferences":["Budget-awareness may need to be optimized separately from task accuracy rather than emerging as a side effect of capability scaling.","The same interval-estimation protocol could be applied to other scarce resources such as wall-clock time or API rate limits.","Deployment systems could expose the agent's current interval bounds to users as a live dashboard rather than a post-run report.","If interval calibration stays low even after training, hybrid human-in-the-loop review of alerts may be required before full autonomy."],"forward_implications":["Agent strength on tasks does not guarantee strength at budget estimation.","Over-optimism causes continued spending on trajectories that will fail.","Early stopping on low-probability trajectories recovers 28-64% of tokens.","SFT+RL training improves early-stop and alert behavior but leaves interval coverage at 47%.","Budget estimation and task success can be trained as partly independent objectives."],"fun_headline_variants":["Agents keep spending despite low success odds","Early stops recover 28-64% tokens on failures","Budget intervals cover just 47% after training","Strong agents not always budget aware r=0.35"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The rollout-replay measurements on the four environments capture the budget-awareness that matters in actual user deployments.","fun_headline_variants_meta":{"raw":{"variants":["Agents keep spending despite low success odds","Early stops recover 28-64% tokens on failures","Budget intervals cover just 47% after training","Strong agents not always budget aware r=0.35"]},"model":"grok-4.3","cost_usd":0.005751,"raw_usage":{"total_tokens":2764,"prompt_tokens":712,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":57512000,"prompt_tokens_details":{"text_tokens":712,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1993,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":712,"tokens_out":59,"duration_ms":13011,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T23:22:39.331513+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Live deployment of the same agents on real user tasks where predicted intervals and alerts are compared against the actual remaining budget at the moment each alert is issued.","supporting_citations":[],"review_version":1}