{"id":"c705905f-9981-4691-bcc4-aee6091dc790","arxiv_id":"2508.11027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Language-model agents fail badly at finding backup plans when a planned function is disabled, even when a correct alternative is guaranteed to exist.","lead":"This paper introduces a benchmark that forces AI agents to recover when a function they planned to use is disabled, even though an equivalent alternative always exists. All five tested models, including GPT-4o and Gemini 2.0 Flash, lost roughly a third to nearly half their accuracy when forced to find a backup plan.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Backup-plan solvability is asserted, not verified: SQL subquery types such as correlated subqueries may yield no executable alternative after error injection, so the reported accuracy drop may partly reflect unsolvable tasks.","rationale":"The benchmark's inferential move from Table 1 to 'agents struggle to formulate and execute backup plans' depends entirely on the guarantee that backup plans remain executable after error injection. The paper supports this guarantee only by an intuition about SQL subqueries and one worked example with an uncorrelated scalar subquery (Section 3.2, Appendix C.1). No automated per-problem verification is reported, and the quality-control pipeline (empty tables, >100-row outputs) does not test subquery type. Correlated subqueries and several other syntactic positions break the claimed two-call equivalence, so the risk is concrete rather than hypothetical. This is the same load-bearing assumption the reader identified; the stress-test sharpens it by naming a mechanism by which the guarantee can fail and a check that resolves it. Other concerns, such as pre-training contamination and prompt sensitivity, are real but less load-bearing: contamination would tend to affect both conditions, and prompt sensitivity does not explain the within-prompt error/no-error gap. If the verification finds unsolvable items and re-running Table 1 without them erases the drop, the central claim would need substantial qualification; if it finds none, the empirical finding stands. The reader's CONDITIONAL verdict remains appropriate pending this check.","tokens_in":23889,"tokens_out":10263,"duration_ms":110568,"concrete_test":"Run a verification script over the released benchmark: for each of the 830 test problems, execute all candidate solution paths (the direct single-call path and every decomposed multi-call path) with the gold argument values under the same function implementations used in evaluation, and compare each output to the reference answer. Report (a) the number of problems with no executable path yielding the reference, and (b) a breakdown by SQL subquery type (correlated, IN, scalar, FROM). If any such problems exist, re-run Table 1 after excluding them and check whether the accuracy decrease persists; if it shrinks to near zero, the central claim is an artifact of unsolvable items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('language agents struggle to formulate and execute backup plans in response to environment feedback') rests on the guarantee that after the first attempted solution function is disabled, a valid alternative function-call sequence still exists for every one of the 830 test problems. The paper asserts this from the Section 3.2 subquery decomposition ('SQL containing subqueries can be solved in at least two ways') and Appendix C.1, but no per-item verification is reported. The construction as documented demonstrates the equivalence only for an uncorrelated scalar subquery with '=' (function_985 + function_986 in C.1). For correlated subqueries, or subqueries in SELECT, FROM, or IN positions, the two-call scratchpad decomposition does not in general preserve the original SQL semantics. For example, a correlated subquery such as 'WHERE salary > (SELECT AVG(salary) FROM employees e2 WHERE e2.dept_id = e1.dept_id)' cannot be precomputed by one argument-free function call and then piped into a second call; the backup path would either return wrong results or fail at runtime. If even a fraction of the 830 items fall into these patterns, the 'external errors' condition contains unsolvable problems, and the 36-45% accuracy drop overstates agent inflexibility. The quality-control section filters empty tables and queries returning more than 100 rows but does not mention filtering or validating subquery types, and the Limitations section does not list this as a caveat. This is the single most load-bearing assumption because the benchmark's stated purpose is to hold solvability constant ('even with the introduction of these failures, we guarantee that the task remains solvable'); if the guarantee fails, the headline result confounds recovery ability with task solvability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Hell-or-High-Water, a benchmark for studying whether language-model agents can recover from external failures such as a function being disabled. The benchmark is built from Spider text-to-SQL queries by parameterizing SQL, wrapping queries as Python functions, and decomposing queries containing subqueries into two alternative solution paths: a direct single-function call and a multi-call backup plan. Agents operate in a CodeAct environment with a large function library and a search tool. The main experiment (Table 1) measures functional correctness on 830 test problems with and without forced external errors, reporting accuracy drops of 36–45% across five models. Additional experiments analyze error types, the effect of search-space size in a closed world, model scaling, the ability to recognize unsolvable tasks, and question underspecification. The paper concludes that language agents struggle to formulate and execute backup plans in response to environment feedback.","tokens_in":24095,"tokens_out":7613,"duration_ms":83696,"significance":"If the benchmark's solvability guarantee is sound, this is a valuable contribution. It provides a reproducible, deterministic evaluation of agentic recovery from external failures, avoiding LLM-based verification and using bootstrapped standard errors. The open-source code and data, the closed-world ablation, and the error taxonomy are useful for the community. The negative result that resilience to external errors does not improve with model size is an important and practically relevant finding. The benchmark design also isolates backup-planning behavior from basic tool-calling skill more cleanly than prior benchmarks, especially in the closed-world condition.","major_comments":[{"comment":"The central claim of Table 1 rests on the assertion in §3.2 that 'SQL containing subqueries can be solved in at least two ways,' so that after the first solution function is disabled a valid backup plan always exists. Appendix C.1 demonstrates this construction only for an uncorrelated scalar subquery in a WHERE '=' clause (function_985 + function_986 versus function_987). Correlated subqueries, and subqueries in SELECT, FROM, or EXISTS positions, cannot in general be precomputed by one argument-free function call and then piped into a second call through the scratchpad mechanism; the backup path would either return wrong results or require additional orchestration that the two-call decomposition does not provide. The quality-control section filters empty results and outputs over 100 rows but does not filter or validate subquery types, and the Limitations section does not mention this caveat. Because unsolvable problems in the external-error condition would inflate the measured accuracy drop and thus overstate agent inflexibility, please report per-problem verification that, after disabling the first attempted solution function, the known backup sequence is executable and produces the reference answer. Filter or fix any violating problems and rerun the main analysis, or report the fraction of items with invalid backups and quantify the sensitivity of the Table 1 results.","section":"§3.2, Appendix C.1"}],"minor_comments":[{"comment":"The description of error injection states that the first function in the solution set that the agent tries is replaced with an error-throwing version. It is unclear whether, in the 'External errors (Y)' condition, every run actually experiences an injected error or only the runs in which the agent attempts at least one solution-set function. If the latter, runs without any injected error are included in the Y condition, which would dilute the measured drop; please clarify and, if necessary, condition the analysis on trajectories that attempted a solution function.","section":"§A"},{"comment":"There is a typo in the quality-control paragraph: 'correpsond' should be 'correspond.'","section":"§3.2"},{"comment":"The search_tools documentation in Appendix D and the prompt in Appendix E.2 list 'query' as the property name but specify 'required': ['search_query'], which does not match. Although the example traces call the tool with 'query', this inconsistency in the provided documentation could confuse models and should be corrected to match the actual API.","section":"Appendix D/E"},{"comment":"The footnote states that 'there will always exist multiple subsets of Fs that lead to a correct solution'; this is exactly the unverified claim that underlies the benchmark. It should be either supported with a per-problem check or qualified with the verification results requested in the major comment.","section":"§A, footnote 10"},{"comment":"The left panel's legend labels 'Distractors only (Y)' and 'Distractors only (N)' are not defined in the text or caption; please explain what the Y/N distinction refers to.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unverified backup-plan solvability guarantee. If the authors can verify all 830 test problems and report the results, the paper's central claim would be on solid footing. The contribution is otherwise solid and within scope for a general NLP/ML venue. I recommend major revision rather than rejection because the issue is fixable by additional validation and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely reusable benchmark for a real failure mode: deterministic external-function failures on paired direct/composed SQL-derived paths, with functional correctness checked against the database. Second, the main number—five models losing 36–45% accuracy when the first function they try is disabled—is direct and believable as a qualitative finding, but it leans on a solvability guarantee that the paper asserts rather than verifies per item. That matters.\n\nIt does a lot right. The construction is automatic and reproducible, function names are scrambled to reduce memorized solutions, docs are unit-tested, and evaluation is deterministic against SQL ground truth. The sweep is thoughtful: open/closed world, search-space size, model scale, underspecification, and even the no-solution case. The error taxonomy (search/ID/chaining/tool use) is useful; the coding procedure for it is not fully specified, which is a minor cost.\n\nThe soft spot that matters: the guarantee that all 830 test problems remain solvable after a failure is based on subquery decomposition, and the appendix only demonstrates the equivalence for an uncorrelated scalar subquery with =. Correlated subqueries and subqueries in SELECT/FROM/IN are not shown to survive the scratchpad method. If even a fraction of backup plans are broken, the reported drop overstates agent inflexibility. This is easy to check deterministically and should be reported. The 'novel relative to pretraining data' claim is asserted, not tested—less worrying given the scrambled names, but still a claim.\n\nThe central qualitative finding, that models find the right tool but fail to adapt to explicit error feedback, is likely to survive verification. But the absolute numbers and the benchmark's meaning as a clean measurement depend on that guarantee. I would send this to a serious referee.","headline":"A genuinely reusable benchmark for a real failure mode, but the headline accuracy drop rests on a solvability guarantee the paper asserts more than it verifies.","tokens_in":24723,"tokens_out":2993,"would_cite":true,"duration_ms":32898,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language agents fail to recover when the first tool in their plan breaks, even with a valid alternative.","keywords":["agentic planning","backup plans","external failures","function calling","LLM agents","fault tolerance","tool use","benchmark"],"falsifier":"After the failure injection, run an exhaustive or oracle search over all 830 test problems using the same documentation the agents see, enumerating every one-step and chained call sequence and checking whether at least one produces the gold answer. If a nontrivial fraction of problems have no executable alternative, the measured accuracy drop partly reflects unsolvable tasks rather than agent inflexibility; if all alternatives work, the drop measures recovery ability directly.","tokens_in":23624,"feed_emoji":"🤖","tokens_out":9088,"duration_ms":88944,"temperature":0.7,"pith_summary":"This paper sets out to measure whether language-model agents can recover when an external failure breaks the first step of their plan. It introduces a benchmark, Hell-or-High-Water, in which every problem has two distinct, non-overlapping sequences of function calls that yield the correct answer, and then automatically disables the first function the agent attempts so that completing the task requires discovering and executing a backup plan. Across commercial and open-weight models, accuracy falls by roughly a third to nearly half when this failure is injected, even though every problem remains solvable. The paper argues that current agents can often identify the right function in context but cannot reliably adapt to environment feedback by reformulating their approach.","feed_headline":"Agent accuracy drops when a tool fails, even with backups","feed_subtitle":"Every task has a working backup path; models still lose a third to half their accuracy when the first tool fails.","key_machinery":"The load-bearing construction is the pairing of two guaranteed solution paths per question, derived from parameterized SQL queries containing subqueries: the full query becomes one Python function giving a direct solution, while the subquery and the outer query become separate functions whose outputs chain together as the backup path. At run time the benchmark takes the first function from the correct set that the agent attempts and replaces it with a version that always raises an error, such as 'function 987 is currently unavailable. Please try a different function.' This makes every episode a forced test of backup planning while keeping the task solvable. The second ingredient is a large function search space of 4,450 tools with deliberately uninformative names, so the agent must consult tool documentation and search results instead of guessing purposes from names.","core_discovery":"The central claim, stated on the paper's own terms, is that language agents are not fault-tolerant planners: when the environment reports that a chosen function is unavailable, they frequently fail to switch to an alternative route even when a correct alternative is guaranteed to exist. The benchmark is built so that each of 830 test questions can be solved either by a single function call or by composing two or more subquery functions, and both routes are known to be valid. In the open-world setting, where agents must search among 4,450 functions with uninformative names, injecting external errors lowers accuracy from 60.5–71.4 percent to 23.3–41.1 percent across the models tested; Gemini 2.0 Flash falls from 71.4 to 41.1 percent and GPT-4o from 60.5 to 38.4 percent. Error analysis attributes the majority of failures to search, meaning the correct tool never appears in results, followed by failure to select the correct tool once it appears. The paper concludes that scaling model size improves absolute performance but does not make agents resilient to external failures.","pith_inferences":["A testable extension is to give agents an oracle search tool that always surfaces the correct backup functions; if accuracy still drops, the bottleneck is plan reformulation rather than retrieval, and if it recovers, retrieval is the lever.","Because the error message is explicit and a valid backup is guaranteed, the benchmark likely measures a lower bound on real-world difficulty: in production, backups may not exist, may be undocumented, or may fail silently, so the recovery gap could be wider outside the benchmark.","The failure taxonomy points to an architectural division of labor the paper does not test: a retrieval module or a planner that pre-computes fallback paths could offload recovery from the LLM.","Since the problems are synthesized from a text-to-SQL corpus, the benchmark inherits that source's underspecification and data quirks; rerunning the pipeline on cleaner or more complex instruction sets could reveal whether the recovery gap changes with task ambiguity."],"forward_implications":["If the finding holds, deployed agents should be treated as brittle under API outages and service changes: a plan that was sound a moment ago can fail through no fault of the agent, and the models tested here do not reliably route around it.","The two-path construction gives the field a reusable way to isolate recovery ability from basic tool-calling skill, because the same problem is measured with and without an injected failure.","Because search failures account for more than half of all errors in the open-world setting, improving how agents formulate queries over large tool sets may yield larger gains than improving low-level reasoning.","Scaling model size alone will not deliver fault tolerance; interventions such as explicit retry instructions, persistent memory of past failures, or training on recovery trajectories are the plausible levers.","Underspecification is separable from recovery: adding disambiguating hints helped some models substantially but did not close the failure gap."],"supporting_citations":[{"why":"Supplies the text-to-SQL questions and databases from which the benchmark generates its functions and dual solution paths.","marker":"Yu et al., 2018"},{"why":"Provides the interactive code-execution loop through which agents call functions and observe outputs and error messages.","marker":"Wang et al., 2024b"},{"why":"Basis of the alternating reasoning-and-acting prompt structure used in all agent evaluations.","marker":"Yao et al., 2023"},{"why":"Defines the Llama-3.1 and Llama-3.3 model families whose performance and scaling behavior are analyzed.","marker":"Dubey et al., 2024"},{"why":"Provides the Qwen-2.5 model family used for the model-size scaling analysis.","marker":"Qwen et al., 2025"}],"fun_headline_variants":["Agents fail to switch tools when the first tool breaks","Tool failures halve agent accuracy, even with backup plans","Language models can't recover when a function fails","Backup plans don't help agents when the main tool errors","Even with guaranteed workarounds, agents refuse to adapt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That every one of the 830 test problems still has a valid alternative sequence of function calls after the first solution function is replaced with an error-raising version, and that this alternative can be carried out using only the provided documentation; the paper asserts this from the SQL-subquery construction but reports no per-problem verification.","fun_headline_variants_meta":{"raw":{"variants":["Agents fail to switch tools when the first tool breaks","Tool failures halve agent accuracy, even with backup plans","Language models can't recover when a function fails","Backup plans don't help agents when the main tool errors","Even with guaranteed workarounds, agents refuse to adapt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2497,"prompt_tokens":1032,"completion_tokens":1465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1385}},"tokens_in":648,"tokens_out":1465,"duration_ms":10343,"temperature":1.0,"reasoning_tokens":1385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:29:08.162071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"After the failure injection, run an exhaustive or oracle search over all 830 test problems using the same documentation the agents see, enumerating every one-step and chained call sequence and checking whether at least one produces the gold answer. If a nontrivial fraction of problems have no executable alternative, the measured accuracy drop partly reflects unsolvable tasks rather than agent inflexibility; if all alternatives work, the drop measures recovery ability directly.","supporting_citations":[{"cited_title":"Narasimhan, and Yuan Cao","cited_arxiv_id":null,"evidence_quote":"Basis of the alternating reasoning-and-acting prompt structure used in all agent evaluations."}],"review_version":2}