{"id":"a8ba02fc-c367-4b58-9b55-575ffc17f9ec","arxiv_id":"2607.20536","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AppWorld-UL perturbs AppWorld's autonomous tasks into 516 user-interaction-requiring tasks; SOTA agents succeed on only 48.6% of them.","lead":"AppWorld-UL adds 516 user-in-the-loop tool-use tasks to the AppWorld simulator, forcing agents to clarify ambiguous goals, report infeasible steps, and ask before costly actions. The strongest tested agent still fails over half of these tasks, flagging user interaction as a key missing capability in current LLM agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"I-TGC relies on an unvalidated LLM mapping from agent questions to K; a systematic mapping bias would inflate the apparent interaction difficulty.","rationale":"The paper is well-engineered: the perturbation methodology is principled, solvability is validated with stubbed solutions, and the oracle/hidden-knowledge ablation provides credible evidence that K is necessary and interaction contributes to difficulty. The simulator-LMM robustness check (Table 3) also speaks to stability. However, the reader's weakest assumption is the right one: the I-TGC metric — the core measure of benchmark difficulty — is computed from the same LLM that generates user responses. If that LLM mis-maps a semantically valid agent question, the agent is both denied the answer and penalized in I-TGC. The paper's manual evaluation of user responses did not verify mapping correctness, only response naturalness. This is not just a nit about metric implementation; it directly affects the headline numbers. If the mapping were validated and found accurate, the concern would be resolved; if not, the apparent difficulty could be an artifact of an overly rigid user. I therefore agree with the reader's conditional verdict and propose a concrete, feasible validation. No other concern — selection of scenarios, metric strictness, or user proactivity — is as directly load-bearing for the paper's central claim.","tokens_in":37379,"tokens_out":11598,"duration_ms":139456,"concrete_test":"Sample 100 agent trajectories across interaction types and compositions. For each supervisor.message call, have two human annotators judge which K (qi, ai) pairs the question semantically targets (or none). Log the simulated user's own mapping (the Listing 1 prompt requires the user to state its chosen pairs in <think> tags). Compute mapping accuracy. For every mis-mapped question, replay the trajectory with the correct K answer substituted for the user's deflection and recompute TGC and I-TGC for the affected models. If mapping accuracy is below ~95% or any headline score shifts by more than 5 points, the benchmark's difficulty and interaction-difficulty claims are not robust to simulator mapping errors; the authors should either harden the mapping (e.g., include a verification step) or report scores under a validated mapping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline metric I-TGC is computed from the simulated user's LLM-based mapping of each agent question to the hand-authored knowledge set K. Section 4.2 says the user 'explicitly determines whether the agent's question maps to any available question–answer pairs,' and this mapping is then used for programmatic evaluation of interaction quality. However, this mapping is not independently validated: the manual evaluation in §6.6 reviewed 312 responses for naturalness ('judging whether a human would respond similarly given K'), not for whether the agent's question was mapped to the correct K pair. The same mapping gates both the agent's access to the answer (if the user fails to recognize a semantically valid question, it deflects) and the I-TGC unit test (the question is counted as unasked). Thus a systematic over-strict mapping would simultaneously suppress task success and inflate apparent interaction difficulty. Table 3 shows low variance across user LLMs, but consistency among LLMs does not establish correctness against ground truth. The central claim that correct user-interaction is crucial for success rests on the gap between Default I-TGC (41.8%) and Oracle TGC (78.1%); if a nontrivial fraction of agent questions are mis-mapped, this gap and the headline 48.6% score are not reliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces AppWorld-UL, a benchmark of 516 user-in-the-loop tasks built by perturbing 34 AppWorld scenarios to require one or more of three interaction types: clarification for underspecified goals, infeasibility communication, and confirmation-seeking. Each task includes a hand-authored user knowledge set K; a constrained LLM-based simulated user answers only questions mapped to K. The paper proposes I-TGC/I-SGC, which add a recall-of-K unit test to AppWorld's programmatic evaluation. In experiments across six LLMs and two scaffolds, the best system (Code Agent with Claude Opus 4.7) reaches 48.6% I-TGC overall and 35.7% on the compositional subset. Ablations show that withholding K drops GPT-5.5 Code TGC to 12.8%, while providing oracle K raises it to 78.1%, supporting the claim that interaction is a major source of difficulty.","tokens_in":37746,"tokens_out":10520,"duration_ms":106648,"significance":"AppWorld-UL is a timely and useful contribution. It moves beyond simple clarification benchmarks by adding infeasibility and confirmation interactions, and it inherits AppWorld's 475-API stateful environment and programmatic evaluation. The perturbation methodology is principled: because K is known by construction, the authors can check solvability with stubbed solutions, simulate a constrained user, and score interaction quality programmatically. The code/data release and the strong oracle/hidden ablation are particular strengths. However, the validity of I-TGC as a measure of interaction quality is not yet established, because it depends on an LLM's mapping from agent questions to K that is not independently validated; this is the main technical risk in accepting the headline numbers.","major_comments":[{"comment":"Both the simulated user's behavior and the I-TGC unit test rely on the same LLM mapping from agent questions to K. The manual evaluation in §6.6 judges naturalness, not mapping correctness; Table 3 shows LLM-user consistency, not accuracy against ground truth. A systematic over-strict mapping would both deflect legitimate agent questions and mark them as unasked, falsely inflating the apparent interaction difficulty. I recommend a human annotation study of mapping correctness on a stratified sample of agent turns, reporting accuracy per interaction type, and a sensitivity analysis in which I-TGC is recomputed under alternative mappings.","section":"§4.2, §6.6; Table 3"},{"comment":"The solvability check verifies that the stubbed solution using all K pairs passes E', but it does not verify that each K pair is necessary—i.e., that no TGC-passing trajectory avoids asking it. Because I-TGC requires all K questions to have been asked, an agent that finds a legitimate alternative solution (AppWorld explicitly allows multiple valid paths) would be penalized even though it completed the task. Please add a per-K-pair necessity check or a meta-evaluation showing that removing any K pair causes the reference solution to fail, and report how many tasks have alternative TGC-passing trajectories without all K questions.","section":"§5.1 (Validating Solvability); §4.1"}],"minor_comments":[{"comment":"Typo: 'takeawy' should be 'takeaway'.","section":"Appendix A"},{"comment":"The phrase 'interaction requirements account for most of the difficulty' is stronger than the evidence, since the Oracle condition changes both the presence of K and its presentation format (auxiliary QA context vs. instruction text). Please qualify the caption.","section":"Figure 5 caption"},{"comment":"The missing GLM 5.1 FC entry is footnoted in the text but should be marked within the table itself to avoid ambiguity.","section":"Table 2"},{"comment":"'Non-conflicting perturbations' is defined only by example; please state the formal condition (e.g., disjoint modifications to S0, I, E, K) used in the automatic check.","section":"§4.1"},{"comment":"The y-axis labels 'avg recall' and 'avg precision' are unclear about the aggregation unit (tasks vs. questions); please define in the caption.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is well-motivated and largely well-engineered; the I-TGC mapping validation is the critical missing piece. If the authors provide the mapping-validation study and the per-K-pair necessity check, I would be willing to accept a revised version. No concerns about novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely useful benchmark, not a hype piece. The perturbation-based transformation of AppWorld's autonomous tasks into user-in-the-loop tasks is a real methodological contribution, and the knowledge-constrained LLM user is a sensible middle ground between rigid rule-based and unconstrained simulators. The headline result—best agent at 48.6% I-TGC, and the ablation showing oracle knowledge raising GPT-5.5 to 78.1% while a non-responsive user drops it to 12.8%—makes a convincing case that interaction is a major bottleneck for current agents. The solvability checks, the manual review of 312 user responses, and the robustness of results across three user LLMs (2.2 I-TGC std) are all to the paper's credit.\n\nThe soft spots are real but not fatal. The I-TGC metric depends on the simulated user's LLM-based mapping of agent questions to the hand-authored K set, and that mapping is not validated against ground truth—the manual review judged naturalness, not whether the right K pair was selected. If the user is systematically over-strict, the metric would both deny the agent answers and count questions as unasked, inflating the apparent interaction difficulty. Consistency across user LLMs doesn't rule this out. That said, the oracle/hidden ablation still holds up: oracle removes interaction entirely and performance jumps to 78.1%, so the central claim that interaction drives difficulty does not rest solely on the mapping.\n\nA smaller issue is that main results have no variance reporting. Claude Opus 4.7 is run at temperature 1, so a few runs could shift the numbers; reporting standard errors or confidence intervals for a benchmark this size would help.\n\nBottom line: this is a serious benchmark that deserves referee time. The authors should add a validation of the user's K-mapping—either a small human-annotated set of agent questions labeled against K, or an analysis of false-negative mappings—and report variance on main results. I'd be happy to cite it once those are addressed. It's a good paper for anyone building or consuming agent benchmarks; the methodological lessons about simulated-user evaluation are worth discussing in a reading group.","headline":"Solid, well-engineered benchmark that extends AppWorld into user-in-the-loop territory; the interaction-difficulty claim holds, but the I-TGC metric's dependence on an unvalidated LLM user mapping is a genuine soft spot that should be fixed before the numbers become canonical.","tokens_in":38132,"tokens_out":2950,"would_cite":true,"duration_ms":33219,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tool-use agents are being tested on the wrong tasks: real requests are vague, infeasible, or risky, and this paper shows that the missing skill — asking the user — is the main bottleneck, with the best agent at 48.6% and oracle knowledge at","keywords":["user-in-the-loop agents","tool-use benchmark","perturbation-based transformation","simulated user","interaction evaluation","clarification and confirmation","AppWorld-UL"],"falsifier":"Have human annotators independently re-score a sample of logged agent–user dialogues, marking for each agent question whether it maps to a K entry and whether the simulated user's answer supplies exactly that entry; recompute I-TGC from these judgments. If agreement with the simulator's mapping is low, or the human-scored I-TGC diverges by more than a few points, the benchmark's interaction metric is not measuring what it claims.","tokens_in":37348,"feed_emoji":"🤖","tokens_out":8067,"duration_ms":84865,"temperature":0.7,"pith_summary":"This paper claims that standard tool-use benchmarks understate the real difficulty of deploying agents because they hand agents complete instructions, while real users give vague, infeasible, or high-stakes requests that require dialogue. It proposes AppWorld-UL, a benchmark of 516 tasks in a stateful nine-app environment, built by a perturbation method: take a well-defined task, remove or alter a specific piece of information, and record the missing information as a user knowledge set K that the agent must elicit. A simulated user, driven by an LLM but constrained to answer only from K, supplies the missing knowledge; a new metric, I-TGC, checks both task completion and whether the agent asked for all of K. The empirical claim is that interaction is the main difficulty: the best system reaches 48.6% I-TGC overall, oracle knowledge raises that to 78.1%, and a user who withholds information drops it to 12.8%.","feed_headline":"Tool-use agents succeed under half the time once users can talk back","feed_subtitle":"A 516-task benchmark adds a simulated user who can clarify, refuse, or approve — the best agent succeeds only 48.6%.","key_machinery":"The central object is the User Knowledge Set K — a set of question-answer pairs containing precisely the information removed from the original autonomous task by perturbation. K does three jobs: it defines the simulated user's knowledge boundaries, it lets the benchmark know exactly what the agent does not know and therefore should ask, and it grounds a programmatic interaction metric by recording which questions the agent actually asked. The perturbation method is the other half: by introducing known gaps into instructions, initial states, and evaluation criteria, the authors create underspecified, infeasible, and confirmation-requiring tasks whose missing information is exactly K.","core_discovery":"The paper's central claim is that the difficulty of user-in-the-loop tool use is measurable and largely attributable to interaction: a state-of-the-art code agent reaches only 48.6% I-TGC overall and 35.7% on compositional tasks, while the same agent given all user knowledge in advance reaches 78.1%, and a user who refuses information drops it to 12.8%. The benchmark formalizes each interactive task as a tuple (I, S0, E, K), where K is the user-knowledge set of question-answer pairs that the agent must elicit through dialogue. The simulated user is constrained to answer only questions that map to K and to deflect others, and the evaluation adds an interaction unit test to the usual goal-comp","pith_inferences":["The recall-only interaction metric deliberately ignores extra questions; an agent that asks many irrelevant questions can still score well if it also asks the required ones. A cost-weighted variant suggested by the paper's own efficiency analysis would reward asking fewer, better questions.","Because K explicitly enumerates what the agent should ask, AppWorld-UL is a natural source of synthetic supervision: successful dialogues show exactly which questions should have been asked, which could be used to train or fine-tune interactive agents.","The paper does not test multi-party knowledge — a user who knows some facts and a colleague who knows others. Since K is a flat set, extending it to per-person knowledge would test whether agents track who knows what.","The observed 'hallucinated self-resolution' failure suggests a sharper evaluation dimension: penalize agents for inventing missing facts rather than asking. A future version could add decoy items that tempt guessing, making the interaction metric more sensitive."],"forward_implications":["Non-interactive success rates will overstate deployed performance: if the agent cannot ask for missing knowledge, it fails even when it can navigate all the APIs.","Interaction skill is separable from tool-use skill: the oracle ablation bounds how much of the gap is attributable to the need to ask, providing a target for future work.","Any stateful benchmark with programmatic evaluation can be turned into a user-in-the-loop benchmark via the same perturbation recipe, so the methodology has leverage beyond AppWorld-UL.","Asking the right questions is strongly diagnostic: on successful tasks agents recall all expected questions, while on failed tasks recall drops to 37–40%, so interaction quality predicts task success.","Compositional tasks — combining clarification, infeasibility, and approval — are disproportionately harder than single-type tasks, so scaling to realistic mixed interactions is the next challenge."],"fun_headline_variants":["Tool-use agents hit 48.6% when users can clarify or refuse","User-in-the-loop benchmark: agents succeed under half the time","With users in the loop, top tool agent succeeds only 48.6%","Talking users make tool-use agents fail 51.4% of the time","Agents that must talk to users solve only 48.6% of 516 tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation's interaction metric depends on the simulated user correctly recognizing when an agent's question maps to one of the hand-authored knowledge-set entries; the paper manually checked 312 responses for naturalness but did not independently validate the correctness of this mapping, so a mis-mapping would unfairly penalize agents that asked the right thing in different words.","fun_headline_variants_meta":{"raw":{"variants":["Tool-use agents hit 48.6% when users can clarify or refuse","User-in-the-loop benchmark: agents succeed under half the time","With users in the loop, top tool agent succeeds only 48.6%","Talking users make tool-use agents fail 51.4% of the time","Agents that must talk to users solve only 48.6% of 516 tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2236,"prompt_tokens":811,"completion_tokens":1425,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1322}},"tokens_in":555,"tokens_out":1425,"duration_ms":12521,"temperature":1.0,"reasoning_tokens":1322,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:29:22.938254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators independently re-score a sample of logged agent–user dialogues, marking for each agent question whether it maps to a K entry and whether the simulated user's answer supplies exactly that entry; recompute I-TGC from these judgments. If agreement with the simulator's mapping is low, or the human-scored I-TGC diverges by more than a few points, the benchmark's interaction metric is not measuring what it claims.","supporting_citations":[],"review_version":1}