Artificial Intelligence · arXiv:2611.04228 v1 · submitted 3 Nov 2026 · 39 references
DualWorldBench: Can Agents Plan Deliveries across Symbolic and Grounded Worlds?
The paper says
“Recent multimodal agents have shown promising progress in embodied planning and web interaction.”
more of the abstract
However, existing evaluations often rely on simplified tasks with limited combinatorial structure, leaving it unclear whether current agents can robustly integrate perception and reasoning for multi-goal planning under temporal, physical, and multimodal constraints. We introduce DualWorldBench, a unified benchmark for evaluating agent planning under controlled symbolic–temporal and grounded–topological complexity. Built around a shared pickup–delivery interface, DualWorldBench consists of three complementary components: TextWorld, which isolates symbolic planning under varying task compositions and release dynamics; SimWorld, which evaluates grounded path planning under controlled topological variation with text, image, or multimodal inputs; and DualWorld-VQA, a diagnostic suite spanning perception, scene understanding, and planning. Experiments reveal a common gap between local understanding and globally consistent planning. In TextWorld, agents struggle to derive optimal task sequences under interacting temporal constraints; in SimWorld, grounded observations are not reliably converted into feasible action sequences. DualWorld-VQA further confirms that perception and topology understanding do not necessarily translate into constrained planning. Together, these results identify the central challenge as composing symbolic or grounded evidence into globally feasible and efficient multi-goal plans. These findings establish DualWorldBench as a challenging testbed for agents with tighter perception–reasoning–planning integration. All datasets and code are publicly available at our project website https://dualworld.netlify.app/.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 4 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
4 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 39 references say.
The paper says, at §5.2.1, Table 1
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
Gemini-2.5-Flash achieves average reward 265.73, exceeding the Online DP baseline (264.36). The paper acknowledges this in the Q1 paragraph ('its reward slightly exceeds that of Online DP') but offers no resolution. Since Online DP is an exact solver over visible tasks (Appendix C.2), no model should surpass it under the same problem instances and reward function.
more of objection 1
This discrepancy directly undermines the central claim that 'even the best-performing models remain unable to produce optimal task plans' and invalidates the quantitative basis for the reward-gap attribution in Figure 4a. The authors must either (a) identify and fix an error in the DP implementation (e.g., penalty rates, priority rules, tie-breaking), (b) confirm that averages are computed over different subsets, or (c) explain why the comparison is valid despite the violation of optimality. Without this resolution, the headline conclusion is unsupported.
The paper says, at §5.1 / Checklist item 7
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.92):
“Answer: [No] Justification: The reported tables and figures contain aggregate results across benchmark instances, but do not report error bars, confidence intervals, or statistical significance tests.”
Pith says: Objection 2 of 4 · major
All reported results lack error bars, confidence intervals, or significance tests. As the authors themselves note in the NeurIPS checklist ('Answer: [No]'), this absence makes it impossible to assess the reliability of the observed differences between models and baselines. Given that the paper's main claims hinge on comparing model reward to DP baselines and on cross-model rankings, basic uncertainty quantification is essential.
more of objection 2
I request that at least one representative experiment (e.g., TextWorld at n=15) be repeated with multiple seeds and reported with standard deviations or confidence intervals.
The paper says, at §5.2.2, Tables 2 and 5
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The 'Feasible' (F) metric is defined as 'the fraction of runs with a successfully evaluated plan and no execution constraint violations.' However, the SimWorld evaluator maps node sequences to coordinates; if a model outputs a syntactically valid but semantically empty sequence (e.g., only START followed by a single delivery), it may still be considered feasible.
more of objection 3
The authors should clarify whether F captures true plan legality or merely format compliance, and whether partially valid plans are counted. This affects the interpretation of high F values for some models.
The paper says, at §3, Equation (4)
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The reward function includes a class-dependent penalty rate λ(c_i). In TextWorld (Appendix A.1), the penalty rates are λ_A=0.1, λ_B=0.2, λ_C=0.3. The Online and Offline DP solvers (Appendices C.2, C.3) use the same rates. However, the benchmark objective J(π) in Equation (5) does not include a term for the negative reward of priority violations (which in Table 1 are scored as R=0, T=9999).
more of objection 4
This inconsistency between the formal objective and the actual scoring rule should be clarified: is the priority violation a hard constraint (plan invalid) or a soft penalty? The paper treats it as an invalidation, which deviates from the maximization of cumulative revenue.
5 minor objections
The paper says, at §5.2.1, Q1 paragraph
Pith says: minor objection 1 of 5
The sentence 'Although its reward slightly exceeds that of Online DP' should be rephrased to acknowledge that this should, in principle, be impossible, and to refer the reader to the resolution (once provided) of the major concern.
The paper says, at Table 1 caption
Pith says: minor objection 2 of 5
The caption says 'Gray shading indicates the best performance' but the gray row is the Offline DP baseline, which is not a model but an oracle. Clarify that the shading marks the reference upper bound, not a model result.
show 3 more
The paper says, at §4.1, 'Planning and Evaluation Interface'
Pith says: minor objection 3 of 5
The phrase 'invokes the planner whenever the task pool changes or the current plan is exhausted' should specify whether the planner is called synchronously or asynchronously. The interruption mechanism (described in Appendix A.1) is important but should be summarized in the main text.
The paper says, at §5.3, Figure 4b
Pith says: minor objection 4 of 5
The pie chart for SimWorld root-cause is informative, but the root-cause ordering (grounding > capacity > state tracking > precedence) is based on a severity hierarchy. Explain the rationale for this ordering and ensure it does not obscure cases where a run has multiple simultaneous violations.
The paper says, at Checklist items 8 and 10
Pith says: minor objection 5 of 5
As noted in the checklist, the paper does not report compute resources or discuss broader impacts. While not blocking for acceptance, these are standard requirements for NeurIPS submissions and should be added.
What would settle it
Reproduce Table 1's average reward for Gemini-2.5-Flash and Online DP by running both on the same set of instances (same seeds, same task configurations) and comparing per-instance rewards. If any instance yields a model reward strictly greater than Online DP, inspect the DP solver's implementation for errors in deadline handling, Type-C priority enforcement, or tie-breaking. Confirm that the DP uses the same reward function (Eq. 4) and same online information set (no future tasks). Report the maximum ratio of model reward to DP reward across instances.
from the skeptic's reading
How the review went
3 readings by deepseek-v4-flash · 4 Nov 2026
- Reader holds up
- Skeptic disagrees: says it holds up only with conditions TextWorld rewards exceeding exact Online DP baseline suggest an evaluation inconsistency that undermines the central claim about models remaining below theoretical optimum.
- Referee major revision: 4 major objections, 5 minor
Not run on this paper: the editor, the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The strongest objection
In §5.2.1 (Table 1), Gemini-2.5-Flash achieves average reward 265.73 while the exact Online DP baseline (described in Appendix C.2 as solving optimally over visible tasks) achieves 264.36. The paper acknowledges this discrepancy ('its reward slightly exceeds that of Online DP', end of Q1 paragraph) but does not resolve it. Since Online DP is an exact solver, no model should surpass it in any instance under the same task set and reward function; if it does, either the DP implementation has an error (e.g., penalty rates, priority rules, or tie-breaking) or the reported averages are computed over different problem subsets (e.g., aggregating across seeds or task counts in a way that favors models). This directly threatens the paper's central claim that 'even the best-performing models remain unable to produce optimal task plans' and the reward-gap attribution in Figure 4a, which assumes the DP baseline is a correct upper bound. Without resolution, the quantitative evidence for the headline conclusion is suspect.
The reader identified the weakest assumption as the benchmark's abstraction (static, deterministic) possibly not transferring to real-world coupling (dynamic obstacles, partial observability). That is a standard limitation for any benchmark and does not undermine the internal validity of the presented experiments. The concern I raise is more fundamental: an apparent contradiction between the paper's own numerical results and the description of the baseline solver. If the Online DP is truly exact, model rewards cannot exceed it; if they can, the comparison is broken, and the central claim that models 'remain below the theoretical optimum' is unsupported by the data. This inconsistency is load-bearing because the paper repeatedly uses the DP baselines as evidence for its main conclusion. It is also actionable: the authors can re-run or clarify the comparison. Therefore I recommend a conditional acceptance pending verification or correction of this discrepancy, rather than unconditional acceptance.
The referee's report
If the benchmark is sound, DualWorldBench provides a valuable and decomposable testbed that distinguishes between failures in scheduling, grounding, and topology-aware planning. The controlled factorial design (Task composition × Release × Modality × Topology) enables targeted diagnosis of where current agents fail. The inclusion of exact DP baselines and a hierarchical VQA suite strengthens the diagnostic power. The public release of code and data supports reproducibility.
Pith's reading of the paper (a machine's summary, not the authors')
DualWorldBench is a new benchmark that systematically evaluates multimodal agents on pickup-delivery planning under both symbolic-temporal and grounded-topological constraints. It decomposes planning into TextWorld for symbolic scheduling, SimWorld for grounded path planning with visual inputs, and DualWorld-VQA for diagnostic probing. Across 14 LLMs and 6 VLMs, the authors find a consistent gap between local task understanding and globally feasible, constraint-consistent multi-goal plans. Even the strongest models fall short of optimal scheduling under online task release and struggle to convert visual observations into executable action sequences. The central challenge identified is the integration of perception, reasoning, and planning into coherent long-horizon decisions.
What carries the argument. The central unifying mechanism is a pickup-delivery task interface with shared precedence, capacity, and reward constraints, instantiated across three complementary benchmarks. TextWorld uses an event-driven symbolic simulator with controlled task composition and release dynamics. SimWorld uses Gazebo-based scenes with procedural and realistic topologies, supporting text, image, and multimodal inputs. DualWorld-VQA provides three-layer diagnostic questions (perception, topology, planning) to decompose failure modes. The shared interface allows controlled variation of symbolic-temporal and grounded-topological difficulty while keeping the evaluation objective fixed.
Core claim. The paper's core discovery is that current state-of-the-art LLM and VLM agents exhibit a systematic failure to compose symbolic or grounded evidence into globally feasible and efficient multi-goal plans as temporal and topological complexity grows. In TextWorld, frontier models approach but remain below theoretical optimal reward, especially under heterogeneous task types and online task release. In SimWorld, weaker models often produce infeasible action sequences, and even stronger models show degraded route efficiency with larger task sets. DualWorld-VQA confirms that intermediate perception and topology understanding do not automatically translate into constrained planning. The paper interprets these results as evidence that the primary bottleneck is not perception or local reasoning but the integration of these into globally consistent planning under combinatorial constraints.
The premise it rests on. The benchmark assumes that a shared pickup-delivery interface with static topology and deterministic dynamics is a sufficient abstraction for evaluating integrated perception-reasoning-planning; if real-world planning involves dynamic obstacles, partial observability, or egocentric viewpoints, the conclusions may not transfer.
If the paper is right (Pith's inferences, not the authors')
The benchmark provides a structured testbed for future research on integrated perception-reasoning-planning systems.
It establishes that scheduling and action sequencing, rather than individual target recognition, are the key bottlenecks in grounded planning.
The results suggest that progress in embodied planning may require dedicated mechanisms for constraint satisfaction and online replanning.
DualWorld-VQA offers a diagnostic tool to isolate whether a model fails at perception, topology understanding, or logical planning, guiding targeted improvement.
The separation of symbolic-temporal and grounded-topological difficulty implies that a modular approach, where a symbolic planner is augmented with a topological cost model, might outperform monolithic end-to-end models. This is not tested in the paper.
The strong performance of text-only planning relative to image-only planning in SimWorld suggests that current VLMs may have underdeveloped spatial reasoning from visual inputs; incorporating explicit spatial representations or navigation graphs could bridge this gap.
The benchmark currently evaluates temporal and topological complexity separately; future work that combines them in the same instances may reveal additional coupling challenges that the paper's separate evaluation cannot capture.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Recent multimodal agents have shown promising progress in embodied planning and web interaction. However, existing evaluations often rely on simplified tasks with limited combinatorial structure, leaving it unclear whether current agents can robustly integrate perception and reasoning for multi-goal planning under temporal, physical, and multimodal constraints. We introduce DualWorldBench, a unified benchmark for evaluating agent planning under controlled symbolic–temporal and grounded–topological complexity. Built around a shared pickup–delivery interface, DualWorldBench consists of three complementary components: TextWorld, which isolates symbolic planning under varying task compositions and release dynamics; SimWorld, which evaluates grounded path planning under controlled topological variation with text, image, or multimodal inputs; and DualWorld-VQA, a diagnostic suite spanning perception, scene understanding, and planning. Experiments reveal a common gap between local understanding and globally consistent planning. In TextWorld, agents struggle to derive optimal task sequences under interacting temporal constraints; in SimWorld, grounded observations are not reliably converted into feasible action sequences. DualWorld-VQA further confirms that perception and topology understanding do not necessarily translate into constrained planning. Together, these results identify the central challenge as composing symbolic or grounded evidence into globally feasible and efficient multi-goal plans. These findings establish DualWorldBench as a challenging testbed for agents with tighter perception–reasoning–planning integration. All datasets and code are publicly available at our project website https://dualworld.netlify.app/.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-04
The authors' main claim, as the classifier picked it out 0.98
“Together, these results identify the central challenge as composing symbolic or grounded evidence into globally feasible and efficient multi-goal plans.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.27 · human participants 0.04 · controlled comparison 0.07 · randomized assignment 0.02 · states quantitative result 0.04 · compares with prior results 0.30 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Recent multimodal agents have shown promising progress in embodied planning and web interaction. | background 1.00 | 0.03 | 0.02 | asserted 0.89 |
| However, existing evaluations often rely on simplified tasks with limited combinatorial structure, leaving it unclear whether current agents can robustly integrate perception and reasoning for multi-goal planning under temporal, physical, and multimodal constraints. | gap 0.98 | 0.07 | 0.03 | asserted 0.93 |
| We introduce DualWorldBench, a unified benchmark for evaluating agent planning under controlled symbolic--temporal and grounded--topological complexity. | contribution 1.00 | 0.88 | 0.05 | asserted 1.00 |
| Built around a shared pickup--delivery interface, DualWorldBench consists of three complementary components: TextWorld, which isolates symbolic planning under varying task compositions and release dynamics; SimWorld, which evaluates grounded path planning under controlled topological variation with text, image, or multimodal inputs; and DualWorld-VQA, a diagnostic suite spanning perception, scene understanding, and planning. | method 0.56 | 0.81 | 0.04 | asserted 1.00 |
| Experiments reveal a common gap between local understanding and globally consistent planning. | result 0.94 | 0.90 | 0.31 | asserted 1.00 |
| In TextWorld, agents struggle to derive optimal task sequences under interacting temporal constraints; in SimWorld, grounded observations are not reliably converted into feasible action sequences. | result 1.00 | 0.91 | 0.16 | asserted 0.99 |
| DualWorld-VQA further confirms that perception and topology understanding do not necessarily translate into constrained planning. | result 0.94 | 0.90 | 0.15 | asserted 0.86 |
| Together, these results identify the central challenge as composing symbolic or grounded evidence into globally feasible and efficient multi-goal plans. | interpretation 0.99 | 0.87 | 0.87 | asserted 1.00 |
| These findings establish DualWorldBench as a challenging testbed for agents with tighter perception--reasoning--planning integration. | interpretation 0.90 | 0.68 | 0.22 | asserted 1.00 |
| All datasets and code are publicly available at our project website https://dualworld.netlify.app/. | contribution 0.75 | 0.37 | 0.02 | asserted 0.99 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-04: 1 request, 7,599 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (11)
Showing 10 of the paper's 11 figures. Figures are the authors', reproduced from the paper.
References (39)
- [1] Mohamed Aghzal, Xiang Yue, Erion Plaku, and Ziyu Yao. Evaluating vision-language models as evaluators in path planning. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6886–6897. IEEE, 2025 on Pith
- [2] Chao Cao, Hongbiao Zhu, Fan Yang, Yukun Xia, Howie Choset, Jean Oh, and Ji Zhang. Au- tonomous exploration development environment and the planning algorithms. In2022 Inter- national Conference on Robotics and Automation, pages 8921–8928, 2022 on Pith
- [3] Yang Chen, Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang, Ming Yang, Yu-Feng Li, and Lan- Zhe Guo.Re 2 Agent: Reflection and re-execution agent for embodied decision making. In NeurIPS 2025 Challenge on Foundation Models for Embodied Agents, 2025 on Pith
- [4] Yang Chen, Lirong Che, Zhenyu Huang, Wenbo Fu, Chuang Wang, Xu Cao, Daqi Liu, Yuzhe Yang, Jian Su, and Lan-Zhe Guo. Harnessvln: Unifying training-free embodied navigation through an agent harness.arXiv preprint arXiv:2609.15195, 2026 on PithPith's reviewarXiv
- [5] Jae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jaehong Kim, and Minsu Jang. Lota- bench: Benchmarking language-oriented task planners for embodied agents.arXiv preprint arXiv:2402.08178, 2024 on PithPith's reviewarXiv
- [6] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, pages 28091–28114, 2023 on Pith
- [7] Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 346–355, 2024 on Pith
- [8] Yining Hong, Rui Sun, Bingxuan Li, Xingcheng Yao, Maxine Wu, Alexander Chien, Da Yin, Ying Nian Wu, Zhecan James Wang, and Kai-Wei Chang. Embodied web agents: Bridging physical-digital realms for integrated agent intelligence.arXiv preprint arXiv:2506.15677, 2025 on PithPith's reviewarXiv
- [9] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. InProceedings of the 62nd Annual Meeting of the Associatio... on Pith
- [10] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Mart´ın-Mart´ın, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference ... on Pith
- [11] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li E Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking llms for embodied decision making.Advances in Neural Information Processing Systems, pages 100428–100534, 2024 on Pith
- [12] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning.Transactions of the Association for Computational Linguistics, pages 635–651, 2023 on Pith
- [13] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023 on PithPith's reviewarXiv
- [14] Lingjun Mao, Jiawei Ren, Kun Zhou, Jixuan Chen, Ziqiao Ma, and Lianhui Qin. Delivery- bench: Can agents earn profit in real world?arXiv preprint arXiv:2512.19234, 2025 on PitharXiv
- [15] Fei Ni, Min Zhang, Pengyi Li, Yifu Yuan, Lingfeng Zhang, Yuecheng Liu, Peilong Han, Longxin Kou, Shaojin Ma, Jinbin Qiao, et al. Embodied arena: A comprehensive, unified, and evolving evaluation platform for embodied ai.arXiv preprint arXiv:2509.15273, 2025. 11 on PitharXiv
- [16] Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. InProceedings of the AAAI Conference on Artificial Intelligence, p... on Pith
- [17] Sophie N Parragh, Karl F Doerner, and Richard F Hartl. A survey on pickup and delivery problems: Part ii: Transportation between pickup and delivery locations.Journal f ¨ur Betrieb- swirtschaft, 58(2):81–117, 2008 on Pith
- [18] Victor Pillac, Michel Gendreau, Christelle Gu ´eret, and Andr ´es L Medaglia. A review of dy- namic vehicle routing problems.European Journal of Operational Research, 225(1):1–11, 2013 on Pith
- [19] Ziqiao Shang, Lingyue Ge, Zi-Jian Cheng, Shi-Yu Tian, Zhenyu Huang, Wenbo Fu, Weiming Wu, Yang Chen, Xiangwen Zhang, Yulan Hu, et al. Maptab: Are mllms ready for multi-criteria route planning in heterogeneous graphs?arXiv preprint arXiv:2602.18600, 2026 on PithPith's reviewarXiv
- [20] Jie-Jing Shao, Bo-Wen Zhang, Xiao-Wen Yang, Baizhi Chen, Si-Yu Han, Wen-Da Wei, Guo- hao Cai, Zhenhua Dong, Lan-Zhe Guo, and Yu-Feng Li. Chinatravel: An open-ended bench- mark for language agents in chinese travel planning.arXiv preprint arXiv:2412.13682, 2024 on PithPith's reviewarXiv
show 19 more
- [21] Jie-Jing Shao, Haiyan Yin, Yueming Lyu, Xingrui Yu, Lan-Zhe Guo, Ivor Tsang, James Kwok, and Yu-Feng Li. Lifting traces to logic: Programmatic skill induction with neuro-symbolic learning for long-horizon agentic tasks.arXiv preprint arXiv:2605.01293, 2026 on PithPith's reviewarXiv
- [22] Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21440– 21455, 2024 on Pith
- [23] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern... on Pith
- [24] Shi-Yu Tian, Zhuo-Xia Wang, Xuan-Yi Zhu, Zhi Zhou, Xinwei Yang, Kun-Yang Yu, Ming Yang, Yang Chen, and Yu-Feng Li. Self-evolving neuro-symbolic skills for tool-augmented spatial reasoning.arXiv preprint arXiv:2608.07955, 2026 on PithPith's reviewarXiv
- [25] Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.Advances in Neural Information Processing Systems, pages 38975–38987, 2023 on Pith
- [26] Bin Wen, Ruoxuan Zhang, Yang Chen, Hongxia Xie, and Lan-Zhe Guo. Aligning progress and feasibility: A neuro-symbolic dual memory framework for long-horizon llm agents.arXiv preprint arXiv:2604.02734, 2026 on PithPith's reviewarXiv
- [27] Qiucheng Wu, Handong Zhao, Michael Saxon, Trung Bui, William Yang Wang, Yang Zhang, and Shiyu Chang. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vlms.arXiv preprint arXiv:2407.01863, 2024 on PithPith's reviewarXiv
- [28] Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents.arXiv preprint arXiv:2402.01622, 2024 on PithPith's reviewarXiv
- [29] Qinghong Xu, Jiaoyang Li, Sven Koenig, and Hang Ma. Multi-goal multi-agent pickup and delivery. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 9964–9971, 2022 on Pith
- [30] Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025. 12 on PithPith's reviewarXiv
- [31] Zhuoqun Xu, Yang Liu, Xiaoqi Li, Jiyao Zhang, and Hao Dong. Human-centered in-building embodied delivery benchmark.arXiv preprint arXiv:2406.17898, 2024 on PithPith's reviewarXiv
- [32] Fan Yang, Chao Cao, Hongbiao Zhu, Jean Oh, and Ji Zhang. Far planner: Fast, attemptable route planner using dynamic visibility update. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 9–16, 2022 on Pith
- [33] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embod- ied agents. InInternational ... on Pith
- [34] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, pages 20744–20757, 2022 on Pith
- [35] Junyan Ye, Dongzhi Jiang, Jun He, Baichuan Zhou, Zilong Huang, Zhiyuan Yan, Hongsheng Li, Conghui He, and Weijia Li. Blink-twice: You see, but do you observe? a reasoning benchmark on visual perception.arXiv preprint arXiv:2510.09361, 2025 on PitharXiv
- [36] Pingyue Zhang, Zihan Huang, Yue Wang, Jieyu Zhang, Letian Xue, Zihan Wang, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, et al. Theory of space: Can foundation models construct spatial beliefs through active exploration? InThe Fourteenth International Confe... on Pith
- [37] Xunyi Zhao, Jian Zhou, Sihao Lin, Gengze Zhou, Zerui Li, Xinyu Yan, Jiajun Liu, Anton van den Hengel, and Qi Wu. NavHarness: Towards lifelong embodied navigation.arXiv preprint arXiv:2609.34276, 2026 on PithPith's reviewarXiv
- [38] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023 on PithPith's reviewarXiv
- [39] Nannan Zhu, Yonghao Dong, Teng Wang, Xueqian Li, Shengjun Deng, Yijia Wang, Zheng Hong, Tiantian Geng, Guo Niu, Hanyan Huang, et al. Cvbench: Benchmarking cross-video synergies for complex multimodal reasoning.arXiv preprint arXiv:2508.19542, 2025. 13 Table of Contents in Appe... on PitharXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment