{"id":"ef449f24-1ddc-40cd-9bb0-a078734bbd10","arxiv_id":"2506.15677","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark integrates AI2-THOR, Google Street View, and functional websites to test agents that must combine physical actions with online information retrieval.","lead":"This paper builds a simulated testbed where AI agents must switch between 3D indoor and street-level outdoor scenes and live websites to complete cooking, navigation, shopping, travel, and geolocation tasks. It reports that current multimodal LLM agents finish only a small fraction of these hybrid tasks, while human volunteers succeed far more often.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human baseline in Section H lacks interface/action-space details; the headline human-vs-AI gap could be inflated by interface asymmetry rather than integration intelligence.","rationale":"The benchmark is a real contribution: it provides a unified environment, public code, and a nontrivial suite of cross-domain tasks, and the absolute AI performance is low even setting humans aside. However, the paper's headline claim is explicitly comparative: 'significant performance gaps between state-of-the-art AI systems and human capabilities.' That comparative claim rests on the Section H human numbers, which are underspecified in exactly the ways that matter for a fair comparison. If humans interacted through a natural GUI while agents emitted discrete commands, then the benchmark measures not only integrated intelligence but also interface adaptability, and the size of the gap cannot be attributed cleanly to cross-domain reasoning. The reader's weakest_assumption identifies this same concern, and my read does not move the verdict: the paper should remain CONDITIONAL until the human protocol is detailed or a constrained-interface human study is run. I also note the secondary limitation that the error-taxonomy claim is based on a single model (GPT-4o) and a single domain (cooking), which should be broadened before 'current models predominantly struggle with cross-domain integration' is treated as a general result.","tokens_in":20182,"tokens_out":4272,"duration_ms":51571,"concrete_test":"Release the full human-study protocol and run a controlled comparison: recruit at least 10 participants per task domain, have them complete the same tasks through the exact same discrete action interface as the LLM agents (same click[id]/type[id]/switch_environment command set and same observations), and report per-task accuracy, number of trials, and completion time. If human accuracy under the constrained interface remains near 77–93%, the gap stands; if it drops substantially toward the model range, interface asymmetry is a major confound and the human-upper-bound claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—SOTA LLM agents score 6.4% vs 77.08% on cooking and 9–35% vs 90%+ on outdoor tasks—depends on treating the Section H human numbers as a fair upper bound. Section H reports only that UCLA CS/Statistics students completed 'the same set of tasks' in 2-hour sessions; it omits the number of participants, trials per task, the exact GUI/interface, the action vocabulary, and whether humans were constrained to the same discrete command set as the models (click[id], type[id], switch_environment, etc.). If humans used natural mouse/keyboard interaction while the LLM agents had to emit low-level discrete commands, part of the 10–70 point gap could reflect interface friction rather than cross-domain integrated intelligence. Because the paper's abstract and §5 use this gap to motivate the entire paradigm, and because the error-analysis conclusion ('cross-domain errors, 66.6%') is computed only for GPT-4o on cooking, the benchmark's central 'humans are far better at integration' claim is not yet securely quantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Embodied Web Agents, a paradigm and benchmark for agents that must jointly act in a 3D embodied environment and use web resources. The authors build a platform combining AI2-THOR for indoor cooking, Google Street View for outdoor navigation, and functional websites for recipes, shopping, maps, and Wikipedia; they construct roughly 1.5k tasks across navigation, shopping, traveling, cooking, and geolocation; and they evaluate GPT-4o, Gemini, Qwen, and Intern models. The main empirical claims are that state-of-the-art LLM agents perform far below humans (e.g., 6.4% vs 77.08% overall accuracy on cooking) and that failures are dominated by cross-domain integration errors rather than single-domain skills.","tokens_in":20502,"tokens_out":3886,"duration_ms":44927,"significance":"If the benchmark construction and human baseline hold up, this is a valuable community resource: it is one of the first testbeds requiring agents to switch between an embodied environment and live web tools, and the public release of environments, code, and websites is a concrete strength. The multi-model evaluation and the detailed error taxonomy also provide actionable starting points. The headline quantitative claims, however, are only as strong as the undocumented human baseline and the geolocation dataset-selection choices, so the central results should be read with caution until those parts are clarified.","major_comments":[{"comment":"The human numbers that anchor the paper's central gap—77.08% cooking, 90.28–92.59% outdoor—are reported without the number of participants, trials per task, the exact interface, or the action vocabulary. If human participants used natural mouse/keyboard and unconstrained clicking while LLM agents were restricted to the discrete command set in Table 1, including the explicit switch_environment action, part of the reported gap could reflect interface asymmetry rather than differences in integrated intelligence. Please either provide a detailed human-protocol appendix or substantially soften the abstract and §5.3 claims that humans are far better at integration.","section":"Appendix H (Human Performance)"},{"comment":"The geolocation dataset is deliberately sampled from FairLocator's Breadth set at coordinates 'where we hypothesize web information may improve prediction accuracy.' This selection can inflate the Table 4 improvement of the embodied web agent over the passive baseline, because the comparison is performed on a subset chosen specifically to favor web access. The paper should evaluate on an unselected sample as well, or explicitly discuss how this selection criterion affects the claimed benefit of web integration.","section":"§4 and Appendix D.2 (Geolocation)"},{"comment":"The error analysis supporting the broad conclusion that 'cross-domain errors dominate' is computed only for GPT-4o on cooking tasks. No breakdown is provided for navigation, shopping, traveling, or geolocation, and no error analysis is provided for Gemini, Qwen, or Intern. As presented, the 66.6% cross-domain figure is evidence about one model on one task family, not about the benchmark as a whole; the abstract and §1 should be reworded or supplemented with additional error analyses.","section":"§5.4 and Figure 5"}],"minor_comments":[{"comment":"The text says outdoor planning uses GPT-4o-mini, while the table header says 'GPT'; please align the notation consistently.","section":"Table 2 and §5.1"},{"comment":"The column layout under 'Vision' and 'Text' is ambiguous because the model names are not visually grouped by modality; label each model column explicitly.","section":"Table 3"},{"comment":"The listed error percentages sum to 89.2%, not 100%; please clarify whether there is a residual category and explain how the reported embodied/web/cross-domain totals are computed.","section":"Figure 5"},{"comment":"There are several typos, including 'dicussed' in §4, 'environe' and 'plage' in Figure 17, and 'V oyager' in §2; a copyedit pass would improve readability.","section":"Throughout"},{"comment":"Please report the number of participants, their demographics, and per-task standard deviations or confidence intervals for human accuracy; the current 'ranging from 77.08% to 92.59%' statement lacks uncertainty information.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a solid contribution and the architecture is well described. The main obstacle is the undocumented human baseline and the geolocation selection bias; both are fixable in revision, so I would not reject. I would ask the authors to supply a detailed human-protocol appendix and to either re-run the geolocation comparison on an unselected subset or clearly bound the selection effect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a genuinely useful benchmark paper, not a paradigm breakthrough. The contribution is a testbed that forces agents to alternate between 3D simulated environments and functional websites, with about 1.5k human-verified tasks; the code and data are released. That alone earns a serious referee.\n\nWhat's actually new: prior web benchmarks are screen-only, and prior embodied benchmarks don't require web grounding. Here tasks force cross-domain switching — cooking from a recipe while managing kitchen state, navigating from OpenStreetMap directions, geolocation via exploration plus Wikipedia queries. The geolocation result (active exploration + web query beats static image prediction across every model) is a clean, model-agnostic finding. The generation pipeline is transparent and the data is released.\n\nSoft spots, in order. First, Section H's human baseline is underspecified: no participant count, no interface, no action vocabulary, no statement that humans used the same discrete command set as the agents. Since the abstract and intro lean on the 77% vs 6.4% gap, the authors need to provide those details or soften the framing. I don't think the gap is an artifact — it's huge and consistent across tasks — but the write-up doesn't currently rule out interface asymmetry. Second, the 'cross-domain integration is the bottleneck' claim rests on one model (GPT-4o) on one task family (cooking). It's plausible and the qualitative examples support it, but it's not a measured property of the whole benchmark. Third, geolocation selection is admittedly biased toward locations where web search helps; that's fine for demonstrating the paradigm but limits the geolocation numbers as a general benchmark. Minor: Table 2 labels the model 'GPT' while the text says GPT-4o-mini.\n\nWho this is for: anyone building agents that combine web access with embodied interaction. The math and data look sound, no circularity. It deserves a serious referee, with revisions addressing the human-baseline details and a broader error analysis.","headline":"A real, release-ready benchmark that opens a new evaluation axis, with human-baseline and error-analysis caveats that should be fixed in revision.","tokens_in":20899,"tokens_out":3787,"would_cite":true,"duration_ms":41519,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that current AI agents fail at tasks requiring both web use and physical action, and that the bottleneck—shown by a new benchmark spanning cooking, navigation, shopping, traveling, and geolocation—is cross-domain…","keywords":["embodied web agents","cross-domain reasoning","LLM agents","benchmark","cooking tasks","geolocation","web navigation","simulation platform"],"falsifier":"A concrete check: run the same cooking and navigation tasks with human participants using exactly the discrete action set and interface given to the agents, and compare their accuracy to the reported human numbers; if the gap from agents closes substantially, the claim that the deficit reflects integrated intelligence rather than interface asymmetry would be weakened.","tokens_in":20017,"feed_emoji":"🧭","tokens_out":7857,"duration_ms":79377,"temperature":0.7,"pith_summary":"The paper introduces the idea of Embodied Web Agents: AI systems that can perceive and act in a 3D world while also browsing and retrieving information from the web, and it argues this combination is a missing, necessary direction in agent research. To test this, the authors build a unified simulation platform that connects an indoor kitchen simulator, outdoor street-view navigation, and functional websites (recipes, shopping, maps, Wikipedia), and release a benchmark of about 1.5 thousand tasks across five domains. They find that current LLM agents complete very few of these tasks end-to-end: the best cooking agent reaches 6.4% overall accuracy versus 77.08% for humans, with similar gaps in the other domains. An error analysis attributes most failures (66.6%) to cross-domain integration problems, such as getting stuck in one environment, switching without acting, or misaligning web instructions with physical actions, rather than to isolated web or embodied skills. The takeaway is that integrated physical-digital agency is a distinct and currently unsolved challenge.","feed_headline":"AI agents score 6.4% when web and physical worlds must mix","feed_subtitle":"A new benchmark shows the real failure is switching between browsing, acting, and grounding","key_machinery":"The central machinery is the benchmark and its environment: a unified platform that couples an indoor household simulator, an outdoor street-view navigation graph built from real-world panoramic imagery of four cities, and a set of functional websites (a recipe site, a shopping site, a map service, and an encyclopedia) into one state space. A single explicit action, switch_environment, moves the agent between the embodied and web sides. Tasks are scored by overall accuracy plus separate web-only and embodied-only accuracies. The argument for cross-domain difficulty rests on the error taxonomy, which labels failures as embodied, web, or cross-domain—stuck in one environment, switching without acting, or instruction-action misalignment—and shows cross-domain errors dominate.","core_discovery":"The paper's central claim is that the bottleneck for web-plus-embodied agents is the integration itself, not the component skills. It demonstrates this with the Embodied Web Agents benchmark, where every task requires both meaningful physical action and meaningful web use, and by measuring current LLM agents against human performance. The results show large, consistent gaps—6.4% versus 77.08% overall accuracy in cooking, and smaller but similar gaps in navigation, shopping, and traveling. An error analysis of the best agent shows cross-domain errors (66.6%) dominate, while embodied-only errors are 14.6% and web-only errors are 8.0%. The authors conclude that embodied web agency is a distinct problem that cannot be addressed by improving isolated web or embodied agents.","pith_inferences":["If the human baseline used a more natural interface (mouse, keyboard, free clicking) than the agents' discrete action set, part of the reported gap could shrink; however, the error taxonomy suggests the cross-domain failure modes would likely persist even with equalized interfaces.","The benchmark's structure could be extended to other domain pairs—for instance, physical robot manipulation with live documentation lookup, or virtual-world agents querying database-backed tools—treating the switch_environment action as a general composition operator.","One testable extension: measure how much of the 'stuck in one environment' failures can be removed by a simple budget rule that forces switching after a fixed number of consecutive actions; a large jump would suggest the remaining gap is partly a policy artifact rather than pure reasoning failure."],"forward_implications":["Future agent evaluation should require both web and embodied components in the same task; measuring web-only or embodied-only accuracy is shown to overstate real capability.","The dominant failure modes give concrete design targets: agents need explicit policies for when to switch environments and for grounding web instructions in physical actions.","The cooking results show structured textual scene descriptions help more than first-person vision for current models, suggesting state representation is a key lever for embodied grounding.","On the geolocation tasks, allowing active exploration plus web queries improves all tested models over passive single-image prediction, indicating the embodied-web setup can boost performance on a classic vision task.","Because cross-domain errors dominate, training or prompting that targets environment-switching behavior could yield outsized gains relative to improving perception or web skills alone."],"supporting_citations":[{"why":"Supplies the web-observation protocol (Set-of-Marks screenshot annotations) used by the vision-based baselines and the platform for executing Wikipedia queries in geolocation.","marker":"[Koh et al., 2024]"},{"why":"Provides the environment formalization and the functional websites (map, encyclopedia) that the web environment adapts, plus the text-based accessibility-tree method for web observations.","marker":"[Zhou et al., 2023]"},{"why":"Provides the indoor simulator whose kitchen scenes, object states, and action set define the cooking tasks.","marker":"[Kolve et al., 2017]"},{"why":"Supplies the passive geolocation baseline (FairLocator) and the Breadth dataset from which the 142 geolocation test coordinates are sampled.","marker":"[Huang et al., 2025]"},{"why":"Defines classic single-image geolocation, the task that the paper reinterprets as an embodied, web-assisted exploration problem.","marker":"Hays and Efros [2008]"}],"fun_headline_variants":["AI bots fail at web+physical tasks: 6.4% vs 77% human","Web+physical mix stumps AI: 6.4% on benchmark","Integration is AI's Achilles heel: 6.4% accuracy","Humans 77%, AI 6.4%: the web-physical gap","Why AI can't cook from web: 6.4% vs 77% human"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported human scores (77.08% to 92.59%) are a fair upper bound for comparison, but the paper's Section H does not describe the human interface, action vocabulary, or number of participants; if humans used natural clicking and typing while agents were limited to the discrete action set, the gap could be inflated by interface asymmetry rather than by differences in integrated intelligence.","fun_headline_variants_meta":{"raw":{"variants":["AI bots fail at web+physical tasks: 6.4% vs 77% human","Web+physical mix stumps AI: 6.4% on benchmark","Integration is AI's Achilles heel: 6.4% accuracy","Humans 77%, AI 6.4%: the web-physical gap","Why AI can't cook from web: 6.4% vs 77% human"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001169,"raw_usage":{"total_tokens":4822,"prompt_tokens":920,"completion_tokens":3902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":3794}},"tokens_in":536,"tokens_out":3902,"duration_ms":27022,"temperature":1.0,"reasoning_tokens":3794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:51:22.032425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: run the same cooking and navigation tasks with human participants using exactly the discrete action set and interface given to the agents, and compare their accuracy to the reported human numbers; if the gap from agents closes substantially, the claim that the deficit reflects integrated intelligence rather than interface asymmetry would be weakened.","supporting_citations":[],"review_version":1}