Pith. sign in

REVIEW 4 major objections 5 minor 63 references

The paper claims that the perceptual evidence a vision-language model gathers by calling external spatial tools—localization, segmentation, depth, pose—can be converted into training supervision, letting a small model reason about space acc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:00 UTC pith:EZV5JO5V

load-bearing objection Genuinely new pipeline, unusually thorough ablations, but the transfer claim is hostage to an unproven train/eval disjointness on BOPASK. the 4 major comments →

arxiv 2607.27703 v1 pith:EZV5JO5V submitted 2026-07-30 cs.AI

SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

Yang Zhou , Zixuan Huang , Sunzhu Li , Zhuo Yang , Chen Zhang , Shunian Chen , Caijun Yan , Jianyao Xu
show 5 more authors
Shunyu Liu Weijie Fu Peiliang Li Xiaozhi Chen Yuxiang Cai
This is my paper
classification cs.AI
keywords spatial reasoningvision-language modelstool-augmented agentscapability internalizationreinforcement learningperceptual distillationcompositional benchmarkingembodied AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a vision-language model can absorb the specialist perceptual skills of external tools by practicing with those tools and then converting its own successful tool-use experiences into training data. It answers yes, with a three-stage recipe: equip the model with spatial tools, fine-tune it to use them well, then verbalize successful tool-use trajectories into reasoning chains and train the model to answer directly without the tools while still retaining tool-use ability. On a new 516-question compositional benchmark, this lifts an 8-billion-parameter model from 35.3% to 72.7% without tools and 91.3% with tools; similar gains appear on an existing spatial benchmark. If correct, the work shows a path toward models that carry specialist perception in their own weights and treat external tools as optional augmentation—useful for embodied agents that must act under limited compute or connectivity.

Core claim

On its own terms, the paper's central claim is that perceptual evidence accumulated during tool interaction can be converted into supervision that lets a VLM internalize specialist perceptual capabilities. The mechanism is trajectory-guided internalization: successful reason-and-act tool-use trajectories are first consolidated turn by turn into evidence units, then globally verbalized into a tool-free perceptual reasoning chain, and finally trained through a dual view—one objective for direct reasoning and one for tool use. The authors report that the resulting model, SpatialCLI-8B, improves from 29.3% to 84.6% with tools and retains 73.8% without tools on MindCube, and reaches 91.3% with to

What carries the argument

The load-bearing machinery is the three-stage Call-Learn-Internalize loop. Call exposes four specialist vision models—localization, segmentation, depth, and pose—as callable tools inside an interleaved reason-and-act agent loop. Learn uses Cold-Start SFT followed by agentic RL with group-relative policy optimization to make the model plan tool calls, parse structured returns, and terminate correctly. Internalize takes successful trajectories and turns them into supervision: a turn-wise evidence consolidator verbalizes each tool result into an evidence unit, a global verbalizer merges the units into a single perceptual reasoning chain, and Dual-View Capability Internalization jointly optimize

Load-bearing premise

The headline numbers stand only if the training pool never overlapped with the evaluation images and if the benchmark answers are genuinely determined by what is visible in the images rather than by the same tool outputs the model is trained to imitate.

What would settle it

Swap the four specialist backends for different but equally accurate implementations at evaluation time, and run a near-duplicate search between the 37,000-task training pool and every evaluation image; if the without-tools score collapses or overlap is found, the claimed internalization is largely memorization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, a VLM can be given specialist perception it did not originally have, simply by practicing with tools and then training on its own successful traces.
  • Tool use and direct reasoning do not have to be traded off: the final model keeps both, so an agent can fall back on built-in skills when tools are unavailable or budget-limited.
  • The recipe should help weaker base models most, since tool augmentation gave the largest gains for the smallest model tested; this suggests a way to upgrade cheaper deployed models.
  • Internalization continues to improve with more trajectory data and larger models even after tool-use performance plateaus, so the bottleneck is converting tool evidence into training data.
  • Compositional spatial questions that require coordinating several capabilities at once expose a failure mode that isolated single-capability tests miss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark answers were retained only in cases where specialist-tool evidence unambiguously supported them, the with-tools comparison may be structurally favorable to this method; an independent audit with answers derived from different tool backends would test this.
  • The same Call-Learn-Internalize loop could be applied to any structured expert—program interpreters, simulators, or retrieval engines—turning external capabilities into model weights, not just spatial perception.
  • The without-tools gains are the strong claim; if they survive a leakage audit, then a self-improving loop is possible: the internalized model can generate new trajectories, verbalize them, and train again, compounding the skill without external tools.
  • A sharper test of internalization would evaluate the model against tool outputs from unseen specialist backends; if the model reproduces only the exact training tools' formats, the capability is tool-specific imitation rather than general spatial understanding.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SpatialCLI, a three-stage framework (Call–Learn–Internalize) that equips a VLM with four specialist spatial tools (Locate, Segment, Depth, Pose), fine-tunes the model to use them via Cold-Start SFT and GRPO-style agentic RL, and then verbalizes successful tool-use trajectories into tool-free CoT supervision to internalize the specialist perceptual capabilities. The authors also introduce SpatialCLI-Bench, a 516-example compositional VQA benchmark, and report large gains: Qwen3-VL-8B-Instruct rises from 29.3% to 84.6% with tools and 73.8% without tools on MindCube, reaching 91.3% with and 72.7% without tools on SpatialCLI-Bench, with additional gains on MMSI, DA-2K, and BOPASK.

Significance. If the reported results are valid, this is a substantial contribution: it provides a concrete, well-ablated recipe for transferring specialist perceptual capabilities into VLM weights, shows that tool use and tool-free reasoning can coexist, and introduces a benchmark for compositional spatial reasoning. The paper also ships code, model weights, and a dataset, and includes extensive ablations (Tables 2–3, 9–12; Figures 3–4) plus a dedicated capability-internalization metric (CII). These strengths make the framework independently useful regardless of the final verdict on the headline numbers. However, the central transfer claim — that the model internalizes specialist perception rather than memorizes evaluation answers — rests on training/evaluation disjointness, which is asserted but not demonstrated, and on the fairness of the new benchmark, which is constructed using the same tools that are used for evaluation.

major comments (4)
  1. [B.1 vs §4.1, Table 1] B.1 lists the shared 37,000-task training pool as including 10,000 tasks from BOPASK-Trajectory and 10,000 from BOPASK-Object-Rearrangement. §4.1 evaluates exactly the 'Trajectory' and 'Object-Rearrangement' subsets of BOPASK. No split, sample-identity check, or deduplication procedure is given. As written, the B.1 assertion that training data have 'no textual or visual overlap with any benchmark evaluated' is contradicted by the use of the same named subsets. This directly threatens the w/o-Tools BOPASK gains in Table 1 (54.6/52.3 vs 25.8/13.3) and the macro-average. The authors must provide per-sample disjointness evidence (e.g., image-hash and text dedup) or remove/relabel these subsets.
  2. [B.1, C.2] The no-overlap claim is also load-bearing for MindCube and SpatialCLI-Bench. The training pool contains 10,000 MindCube-Train tasks, while C.2 states that 195 of the 516 SpatialCLI-Bench examples reuse image data from the existing MindCubeBench evaluation set. Without a demonstrated split or deduplication between MindCube-Train and MindCubeBench, and between those images and the MindCube-derived benchmark samples, the w/o-Tools gains on MindCube (73.8 vs 29.3) and on the MindCube portion of SpatialCLI-Bench could reflect image or answer exposure during SFT, RL, or internalization rather than capability transfer. The paper needs a concrete dedup protocol and a report of how many training images share exact or near-duplicate content with each evaluation set.
  3. [§4.2, Table 1] All headline numbers are reported as point estimates with no error bars, confidence intervals, or significance tests. B.2 says evaluations use temperature 1.0 and three runs without fixed seeds, so run-to-run variance is unknown. Many comparative statements — e.g., 'the best result in each column', 'outperform AlloSpatial on every subset', and the claimed parity or advantage of SpatialCLI-8B over larger models — rely on differences as small as 0.1–1.0 points. The paper should report per-run results or standard deviations, and ideally significance tests for the close comparisons that support the 'consistent gains' narrative.
  4. [C.4–C.5, §4.2] SpatialCLI-Bench is constructed using the same four specialist tools that SpatialCLI is trained to invoke: Gemini 3.1 Pro generates answers conditioned on specialist-tool evidence, candidates are filtered when evidence is missing or ambiguous, and humans validate only examples where the filtered evidence supports a unique answer (71.67% retention). Evaluating models 'w/ Tools' therefore rewards alignment with the exact tool stack that defined the answers, which is structurally favorable to SpatialCLI over models that did not train with that stack. This does not invalidate the benchmark for measuring tool-augmented compositional reasoning, but the authors should (a) test sensitivity by swapping in alternative tool implementations (e.g., a different open-vocabulary detector/segmenter), and (b) report how many benchmark answers would change if tool outputs were noisier or if ambiguous-evide
minor comments (5)
  1. [Abstract/§1] The word 'Learn' is rendered as 'L earn' in the abstract and in the first occurrence of the stage list in §1.
  2. [Table 1] The column header 'ObjRrr' appears to be a typo for 'Object-Rearrangement'.
  3. [B.2] The evaluation details say 'We report the mean over three evaluation runs, without fixing evaluation seeds.' Since the main tables provide no variance information, the authors should either fix seeds and report the single run, or provide run-level values so the reader can assess stability.
  4. [Appendix A, Algorithm 1] The pseudocode line 15 says 'eDSFT ← {Interact(πteach, x, U, B) : x ...}', but the SFT dataset is later filtered by correctness; consider noting the filtering step explicitly in the pseudocode for clarity.
  5. [References] Reference [41] is cited as the implementation of verl, but the reference title is 'HybridFlow: A flexible and efficient RLHF framework.' Please clarify the relationship or update the citation.

Circularity Check

0 steps flagged

No by-construction circularity; the pipeline is a standard distillation-and-fine-tuning recipe. Main risks are data-disjointness and tool-derived benchmark answers, which are validity concerns, not circular reductions.

full rationale

The claimed derivation is an empirical training pipeline: specialist tools provide evidence; SFT/RL train tool use; successful trajectories are verbalized into reasoning chains and used as SFT targets; the final model is evaluated w/ and w/o tools on external benchmarks. None of these stages defines a target in terms of the claimed result. The CII metric defines internalization as similarity to specialist-tool outputs, but it is evaluated on held-out examples and is a direct measurement of the training objective rather than a hidden re-use of the fitted values. SpatialCLI-Bench answers are intentionally generated from specialist-tool evidence; this makes the benchmark tool-grounded but does not force the w/ Tools or w/o Tools results, since all baselines receive the same tools and the internalized model must produce answers without tools. The one serious concern is Appendix B.1/B.2: the 37k-task training pool explicitly includes MindCube-Train and BOPASK-Trajectory/Object-Rearrangement, while evaluation uses MindCube and BOPASK Trajectory/Object-Rearrangement; the paper asserts 'no textual or visual overlap' but provides no split or deduplication evidence. This is a missing-support/leakage risk, not a circular reduction by construction, so it does not raise the circularity score above 2. Self-citations ([19], [29]) are not load-bearing. Overall: no identified circular step.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The framework introduces no invented entities: the four 'spatial tools' are interfaces wrapping existing published models (Grounding DINO, SAM 3, Depth Anything 3, VGGT, etc.), CII is a metric, and SpatialCLI-Bench is a dataset. The load-bearing free parameters are the dual-view weight λ (sensitivity-tested), the BOPASK Chamfer threshold, and the evaluation decoding config. The key axioms are specialist-tool reliability, teacher-model reliability, data disjointness, and evaluation stability.

free parameters (3)
  • λ (Tool-Use View loss weight) = 0.5
    Balances internalization and tool-policy retention in L_CI = L_internal + λ L_agentic (§3.3). F.4 shows λ=1.5 collapses w/o Tools from 72.7 to 62.9; the headline results are sensitive to this hand-chosen value.
  • BOPASK per-sample Chamfer threshold = 0.15
    Eq. 3: score = max(0, 1 - d_CD/0.15). This hand-chosen threshold defines the RL trajectory reward and the evaluation metric for the BOPASK columns, coupling training and evaluation.
  • Evaluation decoding hyperparameters = T=1.0, top-p=0.95, top-k=20, presence penalty=1.5
    §B.2: applied to all models, but presence penalty 1.5 is an unusual hand-chosen setting; scores are reported as means over three seeds-unfixed runs, so the setting materially shapes every Table 1 number.
axioms (4)
  • domain assumption Specialist vision models provide reliable perceptual evidence sufficient to serve both as runtime tools and as benchmark ground truth.
    Invoked in §3.1 and C.4. The framework and all SpatialCLI-Bench answers assume SAM 3, Depth Anything 3, VGGT, and friends are accurate; the paper itself notes monocular depth 'can be noisy' (Box D.3).
  • domain assumption Qwen3.5-397B-A17B generates correct tool trajectories and faithful verbalizations (Ψ_ext and Φ_verb) without inventing evidence.
    §3.2-3.3, B.1: ≈2,000 SFT trajectories and ≈42,000 internalization targets are teacher-produced. Any teacher error or hallucinated verbalization propagates into the trained model's reasoning.
  • domain assumption Training data and evaluation benchmarks are disjoint.
    Asserted in B.1 without a demonstrated deduplication procedure; this is load-bearing for interpreting the w/o Tools gains as capability transfer rather than memorization.
  • domain assumption Three evaluation runs without fixed seeds provide stable enough estimates to compare systems differing by 1-2 points.
    §B.2 reports means over three runs with no variance; Table 1 treats small differences as meaningful. This is an unstated statistical assumption.

pith-pipeline@v1.3.0-daily-deepseek · 47277 in / 26883 out tokens · 256501 ms · 2026-08-01T03:00:19.314039+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning. However, a fundamental capability mismatch remains: general VLMs can reason about the overall task but often miss the visual details that determine success, while specialist vision models can capture those details but cannot translate them into task-level decisions. In this work, we propose SpatialCLI, a framework that teaches VLMs to reason with spatial tools and progressively internalize the specialist perceptual capabilities they provide. SpatialCLI proceeds in three stages: (1) Call exposes specialist vision models as spatial tools to augment the VLM's perception; (2) Learn uses Cold-Start SFT and agentic RL to improve tool use; and (3) Internalize verbalizes successful tool-use trajectories to internalize specialist perceptual capabilities. We further introduce SpatialCLI-Bench, a 516-example benchmark for compositional perception across localization, segmentation, depth, and pose. On MindCube, SpatialCLI raises Qwen3-VL-8B-Instruct from 29.3% to 84.6% with tools, surpassing GPT-5.6 Sol with tools (72.1%), while retaining 73.8% without tools after internalization.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 30 linked inside Pith

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Bop-ask: Object-interaction reasoning for vision-language models

    Vineet Bhat, Sungsu Kim, Valts Blukis, Greg Heinrich, Prashanth Krishnamurthy, Ramesh Karri, Stan Birchfield, Farshad Khorrami, and Jonathan Tremblay. Bop-ask: Object-interaction reasoning for vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16746–16757, 2026

    2026

  3. [3]

    Vlm3: Vision language models are native 3d learners.arXiv preprint arXiv:2605.30561, 2026

    Zhipeng Cai, Zhuang Liu, Yunyang Xiong, Zechun Liu, Vikas Chandra, and Yangyang Shi. Vlm3: Vision language models are native 3d learners.arXiv preprint arXiv:2605.30561, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Sam 3: Segment anything with concepts

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Spatialvlm: Endowingvision-languagemodelswithspatialreasoningcapabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowingvision-languagemodelswithspatialreasoningcapabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  6. [6]

    Volo: A physical orchestrator for open-vocabulary long-horizon manipulation.arXiv preprint arXiv:2606.07723, 2026

    Siyi Chen, Hugo Hadfield, Alex Zook, Mikaela Angelina Uy, Chan Hee Song, Erwin Coumans, Xuning Yang, Faisal Ladhak, Qing Qu, Stan Birchfield, et al. Volo: A physical orchestrator for open-vocabulary long-horizon manipulation.arXiv preprint arXiv:2606.07723, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Spacetools: Tool-augmented spatial reasoning via double interactive rl

    Siyi Chen, Mikaela Angelina Uy, Chan Hee Song, Faisal Ladhak, Adithyavairavan Murali, Qing Qu, Stan Birch- field, Valts Blukis, and Jonathan Tremblay. Spacetools: Tool-augmented spatial reasoning via double interactive rl. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 37109–37120, 2026

    2026

  8. [8]

    Geometrically-constrained agent for spatial reasoning

    Zeren Chen, Xiaoya Lu, Zhijie Zheng, Pengrui Li, Lehan He, Yijin Zhou, Jing Shao, Bohan Zhuang, and Lu Sheng. Geometrically-constrained agent for spatial reasoning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 38689–38699, 2026

    2026

  9. [9]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. InAdvances in Neural Information Processing Systems, volume 37, pages 135062–135093, 2024

    2024

  10. [10]

    S-agent: Spatial tool-use elicits reasoning for spatial intelligence.arXiv preprint arXiv:2606.20515, 2026

    Yalun Dai, Hao Li, Shulin Tian, Runmao Yao, Yuhao Dong, Fangzhou Hong, Zhaoxi Chen, Fangfu Liu, Baoliang Tian, Dingwen Zhang, et al. S-agent: Spatial tool-use elicits reasoning for spatial intelligence.arXiv preprint arXiv:2606.20515, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Gemini 3.1 Pro model card, February 2026.https://deepmind.google/models/model-cards/ gemini-3-1-pro/

    Google DeepMind. Gemini 3.1 Pro model card, February 2026.https://deepmind.google/models/model-cards/ gemini-3-1-pro/

    2026

  12. [12]

    Handal: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions

    Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birch- field. Handal: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11428–11435. IEEE, 2023

    2023

  13. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14953–14962, 2023

    2023

  15. [15]

    Tiger: Tool-integrated geometric reasoning in vision-language models for robotics.arXiv preprint arXiv:2510.07181, 2025

    Yi Han, Enshen Zhou, Shanyu Rong, Jingkun An, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Lu Sheng, and Shanghang Zhang. Tiger: Tool-integrated geometric reasoning in vision-language models for robotics.arXiv preprint arXiv:2510.07181, 2025

    arXiv 2025

  16. [16]

    Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes

    Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. InAsian conference on computer vision, pages 548–562. Springer, 2012. 45

    2012

  17. [17]

    What matters in orchestrating robot policies: A systematic study of hierarchical vla agents.arXiv preprint arXiv:2606.10267, 2026

    Jiaheng Hu, Mohit Shridhar, Caden Lu, Dhruv Shah, Hao-Tien Lewis Chiang, Jie Tan, and Annie Xie. What matters in orchestrating robot policies: A systematic study of hierarchical vla agents.arXiv preprint arXiv:2606.10267, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9590– 9601, 2024

    2024

  19. [19]

    Does your reasoning model implicitly know when to stop thinking?arXiv preprint arXiv:2602.08354, 2026

    Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng, Xuanda Wang, Zhixia Zhang, Hongyan Xie, Songshi Liang, Zehao Chen, Xuefeng Xiao, et al. Does your reasoning model implicitly know when to stop thinking?arXiv preprint arXiv:2602.08354, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.arXiv preprint arXiv:1811.00982, 2018

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.arXiv preprint arXiv:1811.00982, 2018

    Pith/arXiv arXiv 2018

  23. [23]

    Cubify anything: Scaling indoor 3d object detection.arXiv preprint arXiv:2412.04458, 2024

    Justin Lazarow, David Griffiths, Gefen Kohavi, Francisco Crespo, and Afshin Dehghan. Cubify anything: Scaling indoor 3d object detection.arXiv preprint arXiv:2412.04458, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Towards long-horizon embodied agents with tool-aligned vision-language-action models.arXiv preprint arXiv:2605.13119, 2026

    Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong, Yuanzhuo Ding, Zhipeng Zhang, Weixin Li, and Siheng Chen. Towards long-horizon embodied agents with tool-aligned vision-language-action models.arXiv preprint arXiv:2605.13119, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Guava: An effective and universal harness for embodied manipulation.arXiv preprint arXiv:2606.18363, 2026

    Haowen Liu, Xirui Li, Shaoxiong Yao, Peng Shi, Tianyi Zhou, Jia-Bin Huang, Furong Huang, and Jiayuan Mao. Guava: An effective and universal harness for embodied manipulation.arXiv preprint arXiv:2606.18363, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean conference on computer vision, pages 38–55. Springer, 2024

    2024

  29. [29]

    Veriweb: Verifiable long-chain web benchmark for agentic information-seeking.arXiv preprint arXiv:2508.04026, 2025

    Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Jialiang Gao, Heng Zhou, Yunhao Yang, Wendong Fan, et al. Veriweb: Verifiable long-chain web benchmark for agentic information-seeking.arXiv preprint arXiv:2508.04026, 2025

    arXiv 2025

  30. [30]

    Budget-aware tool-use enables effective agent scaling.arXiv preprint arXiv:2511.17006, 2025

    Tengxiao Liu, Zifeng Wang, Jin Miao, I Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, et al. Budget-aware tool-use enables effective agent scaling.arXiv preprint arXiv:2511.17006, 2025

    arXiv 2025

  31. [31]

    Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors

    Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. InAdvances in neural information processing systems, volume 37, pages 68803–68832, 2024

    2024

  32. [32]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shan- tanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Pith/arXiv arXiv 2021

  33. [33]

    Gpt-5.6: Frontier intelligence that scales with your ambition, July 2026.https://openai.com/index/ gpt-5-6/

    OpenAI. Gpt-5.6: Frontier intelligence that scales with your ambition, July 2026.https://openai.com/index/ gpt-5-6/

    2026

  34. [34]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025. 46

    Pith/arXiv arXiv 2025

  35. [35]

    Qwen3.5: Towards native multimodal agents, February 2026.https://qwen.ai/blog?id=qwen3.5

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026.https://qwen.ai/blog?id=qwen3.5

    2026

  36. [36]

    Qwen3.7-Plus: Multimodal agent intelligence, June 2026.https://qwen.ai/blog?id=qwen3.7-plus

    Qwen Team. Qwen3.7-Plus: Multimodal agent intelligence, June 2026.https://qwen.ai/blog?id=qwen3.7-plus

    2026

  37. [37]

    Al- lospatial: Agenticharnessframeworkforspatialreasoninginfoundationmodels.arXiv preprint arXiv:2606.08952, 2026

    ShouweiRuan, BinWang, ZhenyuWu, QihuiZhu, YuxiangZhang, JingzhiLi, YubinWang, andXingxingWei. Al- lospatial: Agenticharnessframeworkforspatialreasoninginfoundationmodels.arXiv preprint arXiv:2606.08952, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in neural information processing systems, volume 36, pages 68539–68551, 2023

    2023

  39. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. InAdvances in Neural Information Processing Systems, volume 36, pages 38154–38180, 2023

    2023

  41. [41]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. InProceedings of the IEEE/CVF international conference on computer vision, pages 11888–11898, 2023

    2023

  43. [43]

    Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026

    HY Team, Xumin Yu, Zuyan Liu, Ziyi Wang, He Zhang, Yongming Rao, Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, et al. Hy-embodied-0.5: Embodied foundation models for real-world agents.arXiv preprint arXiv:2604.07430, 2026

    Pith/arXiv arXiv 2026

  44. [44]

    6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark

    Stephen Tyree, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Jeffrey Smith, and Stan Birchfield. 6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13081–13088. IEEE, 2022

    2022

  45. [45]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025

    2025

  46. [46]

    Locateanything: Fast and high-quality vision-language grounding with parallel box decoding.arXiv preprint arXiv:2605.27365, 2026

    Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, et al. Locateanything: Fast and high-quality vision-language grounding with parallel box decoding.arXiv preprint arXiv:2605.27365, 2026

    Pith/arXiv arXiv 2026

  47. [47]

    Orient anything v2: Unifying orientation and rotation understanding.arXiv preprint arXiv:2601.05573, 2026

    Zehan Wang, Ziang Zhang, Jiayang Xu, Jialei Wang, Tianyu Pang, Chao Du, Hengshuang Zhao, and Zhou Zhao. Orient anything v2: Unifying orientation and rotation understanding.arXiv preprint arXiv:2601.05573, 2026

    arXiv 2026

  48. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in neural information pro- cessing systems, volume 35, pages 24824–24837, 2022

    2022

  49. [49]

    Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.arXiv preprint arXiv:1711.00199, 2017

    Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.arXiv preprint arXiv:1711.00199, 2017

    Pith/arXiv arXiv 2017

  50. [50]

    Vlaser: Vision-language-action model with synergistic embodied reasoning.arXiv preprint arXiv:2510.11027, 2025

    Ganlin Yang, Tianyi Zhang, Haoran Hao, Weiyun Wang, Yibin Liu, Dehui Wang, Guanzhou Chen, Zijian Cai, Junting Chen, Weijie Su, et al. Vlaser: Vision-language-action model with synergistic embodied reasoning.arXiv preprint arXiv:2510.11027, 2025

    arXiv 2025

  51. [51]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025

    2025

  52. [52]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, volume 37, pages 50528–50652, 2024

    2024

  53. [53]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. InAdvances in Neural Information Processing Systems, volume 37, pages 21875–21911, 2024. 47

    2024

  54. [54]

    Mmsi-bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. Mmsi-bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Agentic robot: A brain-inspired framework for vision-language-action models in embodied agents.arXiv preprint arXiv:2505.23450, 2025

    Zhejian Yang, Yongchao Chen, Xueyang Zhou, Jiangyue Yan, Dingjie Song, Yinuo Liu, Yuting Li, Yu Zhang, Pan Zhou, Hechang Chen, et al. Agentic robot: A brain-inspired framework for vision-language-action models in embodied agents.arXiv preprint arXiv:2505.23450, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  57. [57]

    Spatial mental modeling from limited views

    Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. InStructural Priors for Vision Workshop at ICCV’25, 2025

    2025

  58. [58]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. InAdvances in Neural Information Processing Systems, volume 38, pages 113222–113244, 2025

    2025

  59. [59]

    Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    Harness vla: Steering frozen vlas into reliable manipulation primitives via memory-guided agents.arXiv preprint arXiv:2607.08448, 2026

    Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, et al. Harness vla: Steering frozen vlas into reliable manipulation primitives via memory-guided agents.arXiv preprint arXiv:2607.08448, 2026

    Pith/arXiv arXiv 2026

  61. [61]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Roborefer: Towards spatial referring with reasoning in vision-language models for robotics

    Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. InAdvances in Neural Information Processing Systems, volume 38, pages 28404–28481, 2025

    2025

  63. [63]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 48

    2023