{"id":"a4c05aa8-27b0-464a-a0b0-5c41e79739c2","arxiv_id":"2605.28532","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FeasiGen generates infeasible agent tasks via critical tool masking and shows nine models have weak feasibility detection with false continue rates up to 73.9%, improved by multi-agent setups.","lead":"This paper introduces FeasiGen, a pipeline that automatically creates infeasible tasks for tool-using AI agents by identifying and masking critical tools from successful execution traces. A smart generalist might read it to see how current agents waste resources on impossible tasks and why better stopping mechanisms matter for practical AI use.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"FeasiGen's masking of shared critical tools may not guarantee infeasibility due to missing capabilities alone","rationale":"The reader's weakest_assumption directly identifies the same methodological hinge on dataset validity. Full text availability does not resolve the lack of detail on verification protocol or alternative-path checks, so the concern stands and the UNVERDICTED status is appropriate.","tokens_in":1691,"tokens_out":298,"duration_ms":19283,"concrete_test":"Sample 50 constructed infeasible tasks; instruct independent human verifiers (or a separate agent) to attempt completion using only unmasked tools and report success rate; if >10% succeed via alternative strategies, the infeasibility labels are unreliable and the detection metrics must be recomputed on the filtered subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (weak infeasibility detection, false-continue rates up to 73.9%) depends on the constructed dataset consisting of tasks that are infeasible precisely because the masked tools are required. The method extracts traces from multiple agent systems, identifies consistently shared tools, and masks them. This assumes no alternative execution paths exist with the remaining tools. Human verification is reported at 94% accuracy, but without explicit protocol details (e.g., whether verifiers tested for workarounds or only checked necessity of the masked tool), other factors could explain continued execution. The multi-agent improvement observation inherits the same dependency.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims to introduce FeasiGen, an automatic pipeline for constructing infeasible tasks for tool-using agents by extracting tool-calling traces from successful executions across multiple agent systems, identifying consistently shared critical tools, and masking them. Human verification confirms over 94% accuracy for the infeasibility annotations. The work introduces feasibility-aware metrics and evaluates nine models, reporting substantially weak infeasibility detection with false continue rates up to 73.9%, while observing that multi-agent architectures reduce erroneous execution under infeasible conditions.","tokens_in":1825,"tokens_out":448,"duration_ms":21263,"significance":"If the task construction reliably produces infeasible tasks due to missing capabilities with no viable alternatives, the findings would highlight a practically important limitation in current tool-using agents' ability to detect and halt on infeasible tasks, with potential implications for reducing computational costs. The automatic pipeline, multi-system trace extraction, and evaluation across nine models would be notable strengths if the core assumptions hold; the multi-agent improvement observation provides a constructive empirical result.","major_comments":[{"comment":"FeasiGen pipeline: the central claim that masking consistently shared critical tools produces tasks infeasible specifically due to missing required capabilities depends on the assumption that no alternative execution paths exist with the remaining tools. The human verification (reported at over 94% accuracy) does not specify whether verifiers explicitly tested for workarounds or only assessed necessity of the masked tool, which is load-bearing for interpreting the 73.9% false continue rate as evidence of weak detection.","section":"FeasiGen description"},{"comment":"Evaluation section: the reported false continue rate of 73.9% (and related claims across nine models) lacks accompanying error bars, dataset sizes, or full details on how the rate was measured, preventing assessment of statistical reliability and whether post-hoc choices affected the result.","section":"Evaluation results"}],"minor_comments":[{"comment":"The abstract does not report the number of constructed tasks, the specific models or systems used for trace extraction, or the precise definitions of the introduced feasibility-aware metrics.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive feedback. We address each major comment below, indicating planned revisions where appropriate. Our responses focus on clarifying the methodology and strengthening the empirical reporting without overstating the current evidence.","responses":[{"response":"We agree that the assumption of no viable alternatives is central and that the human verification protocol requires clearer documentation. The pipeline extracts traces from multiple distinct agent systems and selects tools that are critical across all of them; this cross-system consistency is intended to reduce the likelihood of system-specific workarounds. However, the verification instructions focused on whether the masked tool was necessary given the original successful trace rather than requiring verifiers to exhaustively enumerate alternative tool combinations. We will revise the manuscript to (1) explicitly describe the verification instructions, (2) report the exact percentage of cases where verifiers were asked to consider workarounds, and (3) add a limitations paragraph acknowledging that exhaustive alternative-path search was not performed. This clarification will be added without altering the reported accuracy figure.","revision_made":"partial","referee_comment":"[FeasiGen description] FeasiGen pipeline: the central claim that masking consistently shared critical tools produces tasks infeasible specifically due to missing required capabilities depends on the assumption that no alternative execution paths exist with the remaining tools. The human verification (reported at over 94% accuracy) does not specify whether verifiers explicitly tested for workarounds or only assessed necessity of the masked tool, which is load-bearing for interpreting the 73.9% false continue rate as evidence of weak detection."},{"response":"We acknowledge that the current manuscript does not include error bars, per-model dataset sizes, or a complete measurement protocol. In the revised version we will add: (i) standard error or standard deviation across three independent runs with different random seeds for each model, (ii) the exact number of tasks evaluated per model (currently aggregated), and (iii) a detailed subsection describing the precise definition and computation of the false-continue rate, including any filtering steps. These additions will allow readers to assess statistical reliability directly.","revision_made":"yes","referee_comment":"[Evaluation results] Evaluation section: the reported false continue rate of 73.9% (and related claims across nine models) lacks accompanying error bars, dataset sizes, or full details on how the rate was measured, preventing assessment of statistical reliability and whether post-hoc choices affected the result."}],"tokens_in":1374,"tokens_out":525,"duration_ms":17386,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main takeaway is that tool-using agents are weak at recognizing when a task cannot be finished with the available tools, with false-continue rates hitting 73.9% on the models they tested, and that multi-agent setups cut down on the wasted steps.\n\nWhat is actually new is the FeasiGen pipeline itself. It pulls successful traces from several agent systems, finds the tools that appear in every trace, and masks those to turn the original task into an infeasible one. Human verification is reported at over 94% accuracy, and they add feasibility-aware metrics to score whether an agent stops early. That construction method is a practical step beyond hand-crafted test cases, and the multi-model evaluation gives a clear picture of the current gap.\n\nThe soft spot is the assumption that masking the shared critical tools always produces infeasibility specifically from missing capabilities. The approach relies on the idea that consistent use across diverse systems means no workaround exists with the remaining tools. Human checks help, but without more detail on whether verifiers actively looked for alternative paths or just confirmed the masked tool was used in the original traces, some tasks might still be solvable another way. That would make the high false-continue numbers partly reflect task construction rather than pure detection failure. The abstract also omits dataset sizes and any error bars on the rates.\n\nThis work is for people building or benchmarking tool-using agents who care about compute waste on impossible tasks. The automatic generation and the empirical numbers on detection weakness are solid enough to justify sending it to referees, even with the need for clearer verification protocol and stats in revision.","headline":"FeasiGen gives a workable automatic way to build infeasible tasks and shows most current agents keep going anyway, but the masking step needs tighter checks on whether alternatives really are blocked.","tokens_in":2298,"tokens_out":406,"would_cite":false,"duration_ms":22752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Tool-using agents rarely detect when tasks are impossible with their tools, continuing up to 73.9 percent of the time.","keywords":["tool-using agents","feasibility awareness","infeasible tasks","multi-agent systems","agent evaluation","tool calling","task construction"],"falsifier":"An experiment showing that agents stop appropriately on a set of infeasible tasks constructed by an independent method that does not rely on masking shared critical tools.","tokens_in":2599,"feed_emoji":"🤖","tokens_out":593,"duration_ms":16851,"temperature":0.7,"pith_summary":"The paper presents FeasiGen, a pipeline that turns solvable tasks into infeasible ones by masking tools that multiple successful agent runs consistently require. Evaluations on nine models show agents have weak ability to recognize these cases and stop early. Multi-agent setups reduce the rate of erroneous continuation compared with single agents. This matters because agents spend compute on long reasoning chains that become pointless when required capabilities are absent. The work supplies both the construction method and new metrics focused on feasibility awareness.","feed_headline":"Agents continue on infeasible tasks up to 74 percent of the time","feed_subtitle":"Masking critical tools creates test cases where single models rarely stop but multi-agent setups perform better.","key_machinery":"FeasiGen pipeline that identifies critical tools consistently shared across successful traces from multiple agent systems and masks them to create infeasible tasks.","core_discovery":"FeasiGen extracts tool-calling traces from successful executions across multiple agent systems, identifies critical tools shared across diverse strategies, and masks them to produce infeasible tasks with over 94 percent human-verified accuracy. When tested, agents exhibit substantially weak infeasibility detection, with false-continue rates reaching 73.9 percent; multi-agent architectures measurably lower erroneous execution under the same infeasible conditions.","pith_inferences":["Explicit feasibility checks before tool calls could be trained or prompted separately from task solving.","The same masking approach might reveal similar awareness gaps for other constraints such as time or cost limits.","Deployment in open environments would benefit from logging how often agents reach tool-missing states."],"forward_implications":["Agents waste substantial compute on reasoning chains for tasks they cannot finish.","Multi-agent designs improve the chance that at least one component recognizes missing capabilities.","Standard success-rate metrics miss the cost of continued execution after infeasibility is evident.","Feasibility-aware stopping rules could be added without changing underlying model capabilities."],"fun_headline_variants":["Agents fail to detect up to 74 percent of infeasible tasks","Single models rarely stop on unfeasible tool tasks","Multi-agent reduces false continues under constraints","Infeasibility awareness lacking in most agent evaluations"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Masking tools that appear consistently in successful traces from multiple systems produces tasks that are infeasible specifically because the required capabilities are missing, as confirmed by 94 percent human verification accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Agents fail to detect up to 74 percent of infeasible tasks","Single models rarely stop on unfeasible tool tasks","Multi-agent reduces false continues under constraints","Infeasibility awareness lacking in most agent evaluations"]},"model":"grok-4.3","cost_usd":0.008111,"raw_usage":{"total_tokens":3672,"prompt_tokens":640,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":81112000,"prompt_tokens_details":{"text_tokens":640,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2978,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":640,"tokens_out":54,"duration_ms":24402,"temperature":1.0,"reasoning_tokens":2978,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T12:18:04.990231+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment showing that agents stop appropriately on a set of infeasible tasks constructed by an independent method that does not rely on masking shared critical tools.","supporting_citations":[],"review_version":1}