REVIEW 2 major objections 1 minor 4 references
Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
T0 review · 2 major / 1 minor · reviewed 2026-06-29 · grok-4.3
Pith's one-line read Tool-using agents rarely detect when tasks are impossible with their tools, continuing up to 73.9 percent of the time.
desk verdict FeasiGen gives a workable automatic way to build infeasible tasks and shows most current agents keep going anyway, but the masking step needs tighter checks on whether alternatives really are blocked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
FeasiGen pipeline that identifies critical tools consistently shared across successful traces from multiple agent systems and masks them to create infeasible tasks.
What would settle it
An experiment showing that agents stop appropriately on a set of infeasible tasks constructed by an independent method that does not rely on masking shared critical tools.
Extended reading notes
Core claim
FeasiGen extracts tool-calling traces from successful executions across multiple agent systems, identifies critical tools shared across diverse strategies, and masks them to produce infeasible tasks with over 94 percent human-verified accuracy. When tested, agents exhibit substantially weak infeasibility detection, with false-continue rates reaching 73.9 percent; multi-agent architectures measurably lower erroneous execution under the same infeasible conditions.
Load-bearing premise
Masking tools that appear consistently in successful traces from multiple systems produces tasks that are infeasible specifically because the required capabilities are missing, as confirmed by 94 percent human verification accuracy.
Editorial extensions
If this is right
- Agents waste substantial compute on reasoning chains for tasks they cannot finish.
- Multi-agent designs improve the chance that at least one component recognizes missing capabilities.
- Standard success-rate metrics miss the cost of continued execution after infeasibility is evident.
- Feasibility-aware stopping rules could be added without changing underlying model capabilities.
Reading between the lines
- Explicit feasibility checks before tool calls could be trained or prompted separately from task solving.
- The same masking approach might reveal similar awareness gaps for other constraints such as time or cost limits.
- Deployment in open environments would benefit from logging how often agents reach tool-missing states.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to introduce FeasiGen, an automatic pipeline for constructing infeasible tasks for tool-using agents by extracting tool-calling traces from successful executions across multiple agent systems, identifying consistently shared critical tools, and masking them. Human verification confirms over 94% accuracy for the infeasibility annotations. The work introduces feasibility-aware metrics and evaluates nine models, reporting substantially weak infeasibility detection with false continue rates up to 73.9%, while observing that multi-agent architectures reduce erroneous execution under infeasible conditions.
Significance. If the task construction reliably produces infeasible tasks due to missing capabilities with no viable alternatives, the findings would highlight a practically important limitation in current tool-using agents' ability to detect and halt on infeasible tasks, with potential implications for reducing computational costs. The automatic pipeline, multi-system trace extraction, and evaluation across nine models would be notable strengths if the core assumptions hold; the multi-agent improvement observation provides a constructive empirical result.
major comments (2)
- [FeasiGen description] FeasiGen pipeline: the central claim that masking consistently shared critical tools produces tasks infeasible specifically due to missing required capabilities depends on the assumption that no alternative execution paths exist with the remaining tools. The human verification (reported at over 94% accuracy) does not specify whether verifiers explicitly tested for workarounds or only assessed necessity of the masked tool, which is load-bearing for interpreting the 73.9% false continue rate as evidence of weak detection.
- [Evaluation results] Evaluation section: the reported false continue rate of 73.9% (and related claims across nine models) lacks accompanying error bars, dataset sizes, or full details on how the rate was measured, preventing assessment of statistical reliability and whether post-hoc choices affected the result.
minor comments (1)
- The abstract does not report the number of constructed tasks, the specific models or systems used for trace extraction, or the precise definitions of the introduced feasibility-aware metrics.
Simulated Author's Rebuttal
We thank the referee for the detailed and constructive feedback. We address each major comment below, indicating planned revisions where appropriate. Our responses focus on clarifying the methodology and strengthening the empirical reporting without overstating the current evidence.
read point-by-point responses
-
Referee: [FeasiGen description] FeasiGen pipeline: the central claim that masking consistently shared critical tools produces tasks infeasible specifically due to missing required capabilities depends on the assumption that no alternative execution paths exist with the remaining tools. The human verification (reported at over 94% accuracy) does not specify whether verifiers explicitly tested for workarounds or only assessed necessity of the masked tool, which is load-bearing for interpreting the 73.9% false continue rate as evidence of weak detection.
Authors: We agree that the assumption of no viable alternatives is central and that the human verification protocol requires clearer documentation. The pipeline extracts traces from multiple distinct agent systems and selects tools that are critical across all of them; this cross-system consistency is intended to reduce the likelihood of system-specific workarounds. However, the verification instructions focused on whether the masked tool was necessary given the original successful trace rather than requiring verifiers to exhaustively enumerate alternative tool combinations. We will revise the manuscript to (1) explicitly describe the verification instructions, (2) report the exact percentage of cases where verifiers were asked to consider workarounds, and (3) add a limitations paragraph acknowledging that exhaustive alternative-path search was not performed. This clarification will be added without altering the reported accuracy figure. revision: partial
-
Referee: [Evaluation results] Evaluation section: the reported false continue rate of 73.9% (and related claims across nine models) lacks accompanying error bars, dataset sizes, or full details on how the rate was measured, preventing assessment of statistical reliability and whether post-hoc choices affected the result.
Authors: We acknowledge that the current manuscript does not include error bars, per-model dataset sizes, or a complete measurement protocol. In the revised version we will add: (i) standard error or standard deviation across three independent runs with different random seeds for each model, (ii) the exact number of tasks evaluated per model (currently aggregated), and (iii) a detailed subsection describing the precise definition and computation of the false-continue rate, including any filtering steps. These additions will allow readers to assess statistical reliability directly. revision: yes
Circularity Check
No circularity detected in dataset construction or evaluation
full rationale
The paper defines FeasiGen as an extraction-and-masking pipeline over successful traces from multiple agent systems, followed by independent human verification (reported at >94% accuracy) to label infeasibility. Feasibility-aware metrics are introduced separately to measure agent stopping behavior. No equations, fitted parameters, or self-citations appear in the provided text that would reduce the central claims (false-continue rates, multi-agent improvement) to the inputs by construction. The derivation chain remains self-contained against external human labels and explicit metric definitions.
Assumptions & free parameters
assumptions (1)
- domain assumption Critical tools can be consistently identified across diverse execution strategies from successful tool-calling traces.
Cite this review
Pith. "Pith review of Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents." pith.science (2026). https://pith.science/paper/5BYYCEVR
@misc{pith2026260528532,
author = {Pith},
title = {Pith review of: Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/5BYYCEVR}},
note = {Machine review of arXiv:2605.28532}
}
read the original abstract
Tool-using agents often incur substantial computational cost due to long reasoning chains and iterative tool usage. In practical scenarios, many tasks become infeasible under constrained tool environments, where the capabilities required for successful task completion are unavailable. Detecting infeasible tasks and stopping execution early can significantly reduce unnecessary execution cost. In this work, we propose FeasiGen, an automatic pipeline for constructing infeasible agent tasks by identifying the critical tools required for successful task completion. Our approach extracts tool-calling traces from successful executions across multiple agent systems, identifies critical tools consistently shared across diverse execution strategies, and masks these tools to automatically transform solvable tasks into infeasible ones. Human verification confirms that the infeasibility annotations for our constructed tasks achieve over 94% accuracy. We further introduce feasibility-aware evaluation metrics for measuring whether agents can recognize infeasible tasks and stop execution appropriately. Extensive evaluations across nine models reveal substantially weak infeasibility detection ability, with false continue rate reaching up to 73.9%. We further observe that multi-agent architectures significantly reduce erroneous execution under infeasible conditions.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
InProceedings of the 24th international conference on Software engineering, pages 467–477
Visualization of test information to assist fault localization. InProceedings of the 24th international conference on Software engineering, pages 467–477. Sanyam Kapoor, Nate Gruver, Manley Roberts, Kather- ine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew G Wilson. 2024. Large language models must be taught to k...
2024
-
[2]
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, and 1 others
Causal inference by using invariant prediction: identification and confidence intervals.Journal of the Royal Statistical Society Series B: Statistical Method- ology, 78(5):947–1012. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, and 1 others. 2024. Toolllm: Facilitating large language model...
2024
-
[3]
Mark Weiser
Workflow mining: Discovering process mod- els from event logs.IEEE Transactions on Knowl- edge and Data Engineering, 16(9):1128–1142. Mark Weiser. 1984. Program slicing.IEEE Transac- tions on software engineering, (4):352–357. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu ...
1984
-
[4]
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094. Seungbin Yang, ChaeHun Park, Taehee Kim, and Jaegul Choo. 2024. Can tool-augmented large language models be aware of incomplete conditions?arXiv preprint arXiv:2406.12307. Shuo Yang, Wei-Lin Chiang,...
Reviewed June 29, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.