Pith. sign in

REVIEW 2 major objections 5 minor 14 cited by

Dense spatial rewards let a 7B vision model rival GPT-4o on spatial reasoning

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:02 UTC pith:ZLLB2PNE

load-bearing objection The controlled dense-vs-sparse RL ablation is genuinely useful, but the supplied abstract overclaims GPT-5/30B/14-benchmark results that the experiments do not report. the 2 major comments →

arxiv 2511.07403 v2 pith:ZLLB2PNE submitted 2025-11-10 cs.CV cs.AIcs.CLcs.LG

SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards

Hunar Batra , Haoqin Tu , Hardy Chen , Yuanze Lin , Cihang Xie , Ronald Clark This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords spatial reasoningscene graph generationreinforcement learningdense rewardGRPOCIoUvisual question answeringmultimodal LLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that vision-language models fail at spatial reasoning not from a lack of data but from a lack of grounded supervision. Training on only 7,587 synthetic scene-graph questions with a reward that forces the model to emit a structured scene graph—objects, bounding boxes, relations—before answering, the authors push a 7B model to 71.2% average accuracy across 12 spatial and real-world benchmarks, a +7.2% gain over its base model. That is nearly double the gain of sparse reward RL (+4.0%) and six times the gain of supervised fine-tuning (+1.2%) on the same data. The broader claim is that structured grounding, taught through reward-aligned reasoning, transfers to out-of-distribution tasks better than imitation or final-answer rewards.

Core claim

The central claim is that integrating scene graph generation directly into the reasoning chain of a multimodal LLM, and training it with a dense lexicographically gated reward via GRPO, produces spatial reasoning that generalizes far better per training sample than existing approaches. The model is not given scene graphs as input; it must produce a question-focused subgraph itself, in a fixed <observe>/<scene>/<think>/<answer> format, and it is rewarded for correctness of the final answer, for matching ground-truth object and relation counts, and—only when the answer is right—for the CIoU between its predicted boxes and the ground truth, after Hungarian matching. On 12 benchmarks the authors

What carries the argument

The key mechanism is a multi-objective reward function with lexicographic gating: format reward (valid JSON scene graph in <scene>), count reward (penalty for deviation from ground-truth numbers of objects and relations), accuracy reward (exact match on multiple-choice answers), and spatial reward (average CIoU between Hungarian-matched predicted and ground-truth boxes). The spatial reward is applied only when the answer is correct, and the whole reward fires only when formatting is valid. This gating exists to prevent reward hacking, which the paper documents: naive spatial rewards make the model emit excessive boxes, and count and region-of-interest filtering are needed to keep it focused.

Load-bearing premise

The evaluation assumes that the measured gains are not inflated by image-level overlap between the 7K training samples (drawn from Visual Genome scene graphs) and the evaluation benchmarks, some of which use the same image sources.

What would settle it

Measure SpatialThinker-7B's accuracy on a subset of CV-Bench and MMVP excluding images that appear (or are near-duplicates of) Visual Genome images, and compare with the reported numbers; if gains over the base model shrink to near zero on the disjoint subset, the claim that dense rewards teach general spatial reasoning is weakened. Alternatively, train the same reward recipe on a scene-graph dataset built from entirely different imagery (e.g., photorealistic renders or indoor scans) and check whether the transfer gains on real-world VQA persist.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Dense spatial rewards nearly double the average gain of sparse-reward RL on the same 7K samples (+7.2% vs +4.0% for 7B), suggesting that process-level supervision is a more effective use of small data than final-answer rewards.
  • Spatial grounding trained on synthetic scene-graph questions transfers to six real-world VQA benchmarks (+5.2% over base 7B), indicating that structured reasoning habits generalize beyond the training distribution.
  • A 7B model trained on 7,587 samples outperforms specialized spatial models trained on 700K–2B samples and approaches or exceeds proprietary models on spatial benchmarks.
  • The data pipeline scales to ~108K samples, so the same recipe can be applied to larger models or broader relation vocabularies without new architecture changes.
  • Lexicographic gating, count penalties, and ROI filtering are jointly necessary; removing any one collapses validation accuracy, which pins down the conditions under which dense spatial rewards work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the 7K training images come from Visual Genome and at least two of the spatial benchmarks (CV-Bench, MMVP) draw on the same image source, part of the in-domain gain could reflect Visual Genome-specific priors rather than general spatial competence. The out-of-distribution claim therefore rests on the six real-world benchmarks; a clean test would retrain on disjoint imagery (e.g., 3D render
  • The reward design—format, count, accuracy, CIoU—is not tied to scene graphs in principle; the same lexicographic gating could supervise other structured outputs such as part hierarchies, layout plans, or action sequences, potentially transferring the data-efficiency benefit to embodied tasks like navigation and manipulation.
  • If the central claim is right, a testable prediction follows: increasing the diversity of scene-graph relation types (e.g., adding temporal or causal relations) should produce corresponding jumps on benchmarks that test those relations, and removing count reward should reproduce the documented reward-hacking collapse even on other datasets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes SpatialThinker, an MLLM post-training method that integrates scene graph generation (SGG) with visual reasoning in a single autoregressive pass, trained with GRPO under a multi-objective dense reward (format, count, accuracy, spatial CIoU) with gating. A 7,587-sample synthetic dataset (STVQA-7K) is generated from Visual Genome scene graphs. The central empirical claim is that on 12 spatial and real-world VQA benchmarks, SpatialThinker-7B improves over the base Qwen2.5-VL-7B by +7.2% average, outperforming SFT (+1.2%) and sparse vanilla GRPO (+4.0%), and surpassing GPT-4o by +3.4% average; analogous gains are reported for the 3B variant. The paper also reports out-of-distribution gains on six real-world benchmarks.

Significance. If the controlled comparisons are reliable, the paper provides valuable evidence that dense, spatially grounded rewards and structured scene-graph reasoning can yield substantial spatial reasoning gains from a small synthetic dataset, and that these gains transfer to broader VQA benchmarks. The experimental design for the central comparison is a clear strength: SpatialThinker, SFT, and vanilla GRPO are trained on the same STVQA-7K data with the same base models and the same full-parameter RL budget, isolating the effect of the dense reward. The claimed release of code, models, and dataset would support reproducibility. However, the headline abstract claims (GPT-5/Claude 4 Sonnet/30B/14-benchmark results) are absent from the experiments, and the evaluation may be affected by train/eval image overlap, so the empirical claims currently outrun the evidence.

major comments (2)
  1. [Abstract vs. Section 4] The abstract states that SpatialThinker-7B matches GPT-5 and outperforms GPT-4o, while SpatialThinker-30B surpasses GPT-5 and Claude 4 Sonnet on average across 14 benchmarks, and reports 3.6× and 1.7× relative gains. Section 4 explicitly evaluates 12 benchmarks (Table 4), and Tables 1–4 contain no GPT-5, Claude 4 Sonnet, or 30B results. The 3.6× and 1.7× ratios are also unsupported: from Table 4, SFT gain ratios are 7.2/1.2=6.0 (7B) and 9.0/3.5≈2.6 (3B); from Table 6, dense-vs-sparse OOD ratios are 5.2/2.7≈1.9 (7B) and 8.5/6.0≈1.4 (3B). The manuscript's own page-1 abstract more modestly claims 'nearly doubling' and 'surpassing GPT-4o,' consistent with the body. This internal inconsistency materially changes the paper's stated contribution and must be resolved.
  2. [Section 3.3 / Section 4] STVQA-7K is synthesized from Visual Genome images, and several evaluation benchmarks (notably CV-Bench and MMVP, and potentially SpatialBench/3DSRBench) are constructed from Visual Genome/COCO imagery. The paper never analyzes whether evaluation images overlap with training images. If they do, the in-domain gains (e.g., CV-Bench 78.2%, MMVP 78.0% in Tables 1–2) may partially reflect image-level familiarity with VG objects and layouts rather than generalized spatial reasoning. Since Table 6's out-of-distribution claim rests on only six real-world benchmarks, an overlap analysis or exclusion of overlapping images is necessary to support the generalization conclusion.
minor comments (5)
  1. [Table 5] The table is labeled a reward ablation, but the last row (+ Filtered Dataset pass@2) changes the training data, not a reward component. Relabel to avoid confounding dataset filtering with reward design.
  2. [Section 3.1, Eq. (4)] The term 'lexicographic gating' is used, but the total reward is a weighted sum with hard gates; count and accuracy are combined additively with weights 0.2 and 0.5, so they are not strictly lexicographic. Please clarify the terminology.
  3. [General] No confidence intervals or multiple-seed results are reported for the main comparisons. Given the small training set (7K) and short training (75 steps), the stability of the +3.2% average gain over vanilla GRPO should be assessed or at least discussed.
  4. [Section 3.3] The dataset size is described inconsistently as 7,587, 'top 7.5K', and '7K'; clarify the final count after filtering.
  5. [Title / Section 1] The title says 'Reinforcing3D Reasoning' but the method uses 2D bounding boxes and RGB-only inputs; consider rewording to 'spatial reasoning' to avoid confusion.

Circularity Check

0 steps flagged

No circularity by construction; the headline numbers are measured on external benchmarks, though the abstract overclaims evidence and data-overlap risks remain.

full rationale

The load-bearing derivation chain is: STVQA-7K samples are synthesized from Visual Genome scene graphs (§3.3, App. A); the dense reward R_total is defined as a function of those ground-truth subgraphs and answers (§3.1); GRPO optimizes that reward (§3.2); and the paper's headline results are average accuracies on 12 external benchmarks (Tables 1–4). None of the reported equations define the evaluation metrics in terms of the training labels: R_total is a training objective, not a prediction target on the test benchmarks. The reward ablation in Table 5 is on STVQA-7Kval and is used for reward design only, not as evidence of external generalization. Reward hyperparameters (weights, λs, β) are tuned on the validation split, which is standard model selection and does not make the 12-benchmark gains fitted predictions. The only self-citations (e.g., VLAA-Thinker, Chen et al. 2025a) appear as baselines or as a supporting citation for sparse-reward RLVR; they are not load-bearing for the dense-reward claim. GRPO, CIoU, and lexicographic gating are cited to external standard references. Flagged as non-circular concerns: (1) the Abstract claims GPT-5/Claude 4/30B/14-benchmark results that are absent from Section 4's '12 diverse spatial understanding and real-world VQA benchmarks' and from Tables 1–3—an evidence gap, not a tautology; (2) STVQA-7K is built from Visual Genome while CV-Bench and MMVP may draw on Visual Genome/COCO imagery, a potential contamination/OOD confound the paper does not analyze; (3) the Abstract's '3.6× larger gains' ratio does not match Table 4 (+7.2/+1.2 = 6.0×). These are correctness and verifiability issues, not circularity by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 1 invented entities

The central claim rests on: (i) the correctness of 7,587 synthetic QA labels, guaranteed only by LLM-generated questions plus an LLM judge filter; (ii) the validity of Visual Genome scene graphs as ground truth for count, relation, and CIoU supervision; (iii) a chain of hand-tuned reward weights (w_* and λ_*) selected on the filtered validation split; and (iv) the assumption that benchmark images do not overlap the VG training distribution in a way that inflates OOD claims. No new physical entities are introduced; the only new 'entity' is the question-focused scene-subgraph output format, whose quality is never directly measured.

free parameters (5)
  • Reward weights w_format=0.1, w_count=0.2, w_accuracy=0.5, w_spatial=0.2 = 0.1 / 0.2 / 0.5 / 0.2
    Hand-tuned via the reward ablation loop on the filtered STVQA-7Kval split (§3.1, Appendix C); they determine the relative strength of the four training signals.
  • Count-reward mixing λ_obj=0.7, λ_rel=0.3 = 0.7 / 0.3
    Set by hand in §3.1 to weight object-count vs relation-count fidelity in the count reward.
  • Hungarian matching cost weights λ_spatial=1.0, λ_semantic=2.0 = 1.0 / 2.0
    Hand-set in §3.1; balances box overlap vs label similarity when matching predicted to ground-truth objects for the CIoU spatial reward.
  • KL penalty β = 10^-2 = 0.01
    Selected by ablation on CV-Bench (Appendix D, Table 7); other values (no KL, chi-squared) reduced accuracy.
  • Dataset filtering thresholds (top-10K rated, pass@2 consistency, 7.5K final) = 56,224 → 10,000 → 7,587
    Curation choices in §3.3/Appendix A; the pass@2 filtering adds +13.0 val accuracy in Table 5, showing the filter itself drives the in-distribution metric.
axioms (7)
  • domain assumption Visual Genome human-annotated scene graphs (objects, boxes, predicates) are correct ground truth.
    All STVQA-7K supervision — counts, relations, CIoU spatial reward targets — derives from VG annotations (§3.3, Appendix A).
  • domain assumption GPT-4o pass@2 consistency filtering identifies correct QA labels.
    The dataset keeps only samples where GPT-4o agrees with the Claude-generated label; the filter is not validated against human judgments (Appendix A).
  • domain assumption Exact string match on multiple-choice answers is a faithful accuracy reward.
    The accuracy reward R_a and all 12 benchmark metrics use exact match on MCQs (§3.1, §4).
  • ad hoc to paper Scene-graph structure (objects + bounding boxes + relation triplets) is the right intermediate representation for spatial reasoning.
    The design premise of the method; the paper shows it works but never directly measures whether emitted scene graphs are correct, so it is a postulate validated only through end-task accuracy.
  • standard math GRPO is a valid policy optimizer for this setting.
    Adopted from DeepSeekMath/DeepSeek-R1 (Shao et al., 2024; DeepSeek-AI, 2025) as the RL machinery (§3.2).
  • standard math CIoU provides a dense, meaningful spatial gradient signal.
    Uses Zheng et al. (2020) CIoU for non-overlapping boxes (§3.1); standard in detection, assumed to transfer to reward shaping.
  • domain assumption Evaluation images do not overlap the Visual Genome training distribution in a way that inflates gains.
    Unstated in §4; CV-Bench and MMVP draw on VG/COCO imagery, and no overlap analysis is provided.
invented entities (1)
  • Question-focused scene subgraph output (the <scene> JSON block with object IDs, pixel boxes, relation triplets inside the reasoning chain) no independent evidence
    purpose: Forces the model to emit a structured, localized mental scene graph before answering; used as the target of count and CIoU spatial rewards.
    The paper never evaluates the correctness of the emitted scene graphs against external SGG benchmarks or human annotation; the only evidence is end-task accuracy, so the 'grounded reasoning' mechanism is asserted, not directly demonstrated.

pith-pipeline@v1.3.0-alltime-deepseek · 29358 in / 22223 out tokens · 211273 ms · 2026-08-03T23:02:16.540602+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language tasks, but continue to struggle with spatial reasoning. Existing spatial MLLMs rely on large-scale datasets, explicit 3D inputs, architecture-specific modifications, or sparse Reinforcement Learning (RL) methods that provide insufficient guidance for spatially-grounded reasoning. We introduce SpatialThinker. To our knowledge, it is the first MLLM unifying Scene Graph Generation (SGG) and visual reasoning in a single pass via online RL. The model simulates human-like spatial perception by constructing a mental scene graph of task-relevant objects and relations, and reasoning toward an answer via dense spatial rewards. Our contributions are threefold: (1) SGG-grounded reasoning: integrating SGG directly within the reasoning chain rather than as a disjoint preprocessing step; (2) STVQA-7K: a high-quality spatial VQA training dataset via a scalable synthesis pipeline; and (3) a dense spatial reward design that enforces structured grounding during RL and generalizes to improve broad visual perception. SpatialThinker-7B achieves 3.6$\times$ larger gains over SFT and $1.7\times$ better in- and out-of-distribution generalization than sparse RL. Trained on only 7K samples, SpatialThinker-7B matches GPT-5 and outperforms GPT-4o, while SpatialThinker-30B surpasses both GPT-5 and Claude 4 Sonnet on average across 14 spatial and real-world benchmarks, demonstrating that structured spatial grounding with reward-aligned reasoning enables robust spatial understanding with limited data.

Figures

Figures reproduced from arXiv: 2511.07403 by Cihang Xie, Haoqin Tu, Hardy Chen, Hunar Batra, Ronald Clark, Yuanze Lin.

Figure 1
Figure 1. Figure 1: Method overview of SPATIALTHINKER. Our framework integrates structured scene-graph grounded reasoning with multi-objective dense RL to enhance 3D spatial understanding in multimodal large language models. Recently, reinforcement learning with verifiable rewards (RLVR) has demonstrated superior general￾ization over supervised fine-tuning (SFT) by learning diverse reasoning strategies rather than static patt… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of QA types in STVQA-7K. The dataset spans a diverse range of spatial reasoning skills, covering spa￾tial relations, localization, existence, reach, depth, distance, size, count and orientation. To facilitate reward-aligned spatial reasoning, we con￾struct STVQA-7K, a synthetic visual question an￾swering (VQA) dataset built from human-annotated scene graphs in Visual Genome (Krishna et al., 20… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison between GPT-4o and SpatialThinker-7B. GPT-4o often fails to distinguish objects at a 3D relational level—for example, confusing spatial relations such as beside, behind, and in front of, or missing fine-grained object details. Evaluation Setting. All models are evaluated in a zero-shot setting using greedy decoding (temper￾ature = 0.0). Models default prompting format is used where a… view at source ↗
Figure 4
Figure 4. Figure 4: STVQA-7K dataset construction pipeline. Synthetic Question Generation. Visual Genome scene graphs serve as our foundational ground truth, providing object categories, bounding boxes, and relational triplets for over 150,000 images. We synthetically generate question-answer pairs for a given scene graph data using Claude Sonnet 4 (Anthropic, 2025), synthesizing multiple-choice questions based on the salient… view at source ↗
Figure 5
Figure 5. Figure 5: Examples of generated QA pairs across the nine spatial reasoning categories in STVQA-7K. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: RL training dynamics of SPATIALTHINKER. All reward components (a–d) improve consistently, reflecting stable optimization. Response length (e) shows a non-monotonic trend, indicating emergent reasoning strategies. C REWARD DESIGN PROCESS This section details our approach to designing a robust reward system that guides models toward genuine spatial reasoning while preventing degenerate solutions. Our reward … view at source ↗
Figure 7
Figure 7. Figure 7: Additional qualitative comparisons between GPT-4o and SpatialThinker-7B. SpatialThinker-7B shows stronger spatial grounding and fine-grained object distinction, accurately identifying 3D relations that GPT-4o often confuses. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving

    cs.CV 2026-05 unverdicted novelty 7.0

    DriveSpatial benchmark shows the strongest of 15 VLMs trails humans by 28.4 points on spatiotemporal tasks, with cognitive scene construction as the primary weakness.

  2. DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving

    cs.CV 2026-05 unverdicted novelty 7.0

    DriveSpatial benchmark shows the best of 15 VLMs trails humans by 28.4 points on spatiotemporal driving tasks, with cognitive scene construction as the main failure mode.

  3. ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    ViSRA boosts MLLM 3D spatial reasoning performance by up to 28.9% on unseen tasks via a plug-and-play video-based agent that extracts explicit spatial cues from expert models without any post-training.

  4. SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence

    cs.CV 2025-05 conditional novelty 7.0

    Presents SpatialScore benchmark for MLLM spatial reasoning, evaluates 49 models showing large human gap, and supplies SpatialCorpus plus SpatialAgent to improve performance.

  5. GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement

    cs.GR 2026-07 conditional novelty 6.5

    GReFEM shows MLLMs zero-shot isolate load-activated geometric features for volumetric mesh refinement with higher precision than matched-budget geometric heuristics.

  6. VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations

    cs.CV 2026-03 conditional novelty 6.5

    Sparse multi-view reasoning is largely unsolved for VLMs; grounded CoT with visual evidence improves moderate cases and transfers to real data, but deep multi-hop reasoning still scales poorly.

  7. Geo3R: Mitigating Spatial Reasoning Hallucination in Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0

    Augmenting MLLMs with structured 3D geometric cards from monocular depth, camera calibration, and object orientation reduces spatial reasoning errors on 18 tasks, with gains up to 10.9 points over unaugmented baselines.

  8. GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding

    cs.CV 2026-07 conditional novelty 6.0

    GeoAnchor decomposes 3D spatial reasoning into position, direction, and geometry latent tokens and reports state-of-the-art scores on SPAR-Bench, SPBench, and ViewSpatial with a 2B model.

  9. OneCanvas: 3D Scene Understanding via Panoramic Reprojection

    cs.CV 2026-06 unverdicted novelty 6.0

    OneCanvas aggregates multi-view 3D patches onto one panoramic canvas with continuous angular placement and 3D embeddings, enabling pretrained VLMs to achieve SOTA on SQA3D and VSI-Bench with an order of magnitude less...

  10. Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    VLMs possess a latent 3D scene topology subspace corresponding to Laplacian eigenmaps that can be causally shaped via Dirichlet energy regularization to improve spatial task performance by up to 12.1%.

  11. SpatiO: Adaptive Test-Time Orchestration of Vision-Language Agents for Spatial Reasoning

    cs.CV 2026-04 unverdicted novelty 6.0

    SpatiO uses heterogeneous vision-language agents with test-time orchestration to dynamically weight their contributions for improved spatial reasoning on benchmarks like 3DSRBench and CV-Bench.

  12. OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping

    cs.CV 2026-07 unverdicted novelty 5.0

    OmniView-Space framework with MPSM, tool-guided reasoning, and distillation achieves SOTA on spatial reasoning benchmarks for MLLMs while reducing external geometry dependencies.

  13. Rethinking VLM Representation for VLA Initialization

    cs.CV 2026-05 unverdicted novelty 5.0

    Experiments indicate original VLM representations are crucial for VLA performance, LoRA outperforms full finetuning, and staged robot-data pretraining yields the strongest initialization.

  14. SpatiO: Adaptive Test-Time Orchestration of Vision-Language Agents for Spatial Reasoning

    cs.CV 2026-04 conditional novelty 5.0

    A heterogeneous multi-agent framework with Bayesian test-time orchestration improves spatial reasoning across four benchmarks without parameter updates.

Reference graph

Works this paper leans on

61 extracted references · 40 linked inside Pith · cited by 12 Pith papers

  1. [2]

    Spatialbot: Precise spatial understanding with vision language models.arXiv preprint arXiv:2406.13642,

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models.arXiv preprint arXiv:2406.13642,

    Pith/arXiv arXiv

  2. [4]

    The final set consists of 50% samples from the relation category, and the remaining 50% distributed across the eight other categories

    Following consistency filtering, we retain 6,895 training samples and 692 validation samples ( 75%), indicating high label reliability. The final set consists of 50% samples from the relation category, and the remaining 50% distributed across the eight other categories. To prevent positional bias, answers are uniformly distributed across options A, B, C, ...

    2025

  3. [5]

    Mm-spatial: Exploring 3d spatial understanding in multimodal llms.arXiv preprint arXiv:2503.13111,

    Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, et al. Mm-spatial: Exploring 3d spatial understanding in multimodal llms.arXiv preprint arXiv:2503.13111,

    Pith/arXiv arXiv

  4. [6]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Dama...

    Pith/arXiv arXiv

  5. [7]

    Kimi Team Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao 12 Ding, Hao-Xing Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao...

    Pith/arXiv arXiv

  6. [8]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive.ArXiv, abs/2404.12390,

    Pith/arXiv arXiv

  7. [10]

    Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh

    Jensen Gao, Bidipta Sarkar, F. Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation.2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12462–12469,

    2024

  8. [11]

    Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Ramalingam Chellappa, Chuang Gan, Celso de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull. Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning.2024 IEEE Interna...

    2024

  9. [12]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  10. [13]

    Scene graph reasoning for visual question answering.ArXiv, abs/2007.01072,

    Marcel Hildebrandt, Hang Li, Rajat Koner, V olker Tresp, and Stephan Günnemann. Scene graph reasoning for visual question answering.ArXiv, abs/2007.01072,

    Pith/arXiv arXiv 2007

  11. [14]

    Tenenbaum, and Chuang Gan

    Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9202–9212, 2023a. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Inj...

    Pith/arXiv arXiv

  12. [15]

    Visual language maps for robot navigation.2023 IEEE International Conference on Robotics and Automation (ICRA), pp

    Chen Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation.2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 10608– 10615,

    2023

  13. [18]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsc...

    Pith/arXiv arXiv

  14. [19]

    What’s" up" with vision-language models? investigating their struggle with spatial reasoning.arXiv preprint arXiv:2310.19785,

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s" up" with vision-language models? investigating their struggle with spatial reasoning.arXiv preprint arXiv:2310.19785,

    Pith/arXiv arXiv

  15. [20]

    Vr-gpt: Visual language model for intelligent virtual reality applications.ArXiv, abs/2405.11537,

    Mikhail Konenkov, Artem Lykov, Daria Trinitatova, and Dzmitry Tsetserukou. Vr-gpt: Visual language model for intelligent virtual reality applications.ArXiv, abs/2405.11537,

    Pith/arXiv arXiv

  16. [22]

    Relation-r1: Progressively cognitive chain-of-thought guided reinforcement learning for unified relation comprehension

    Lin Li, Wei Chen, Jiahui Li, Kwang-Ting Cheng, and Long Chen. Relation-r1: Progressively cognitive chain-of-thought guided reinforcement learning for unified relation comprehension. arXiv preprint arXiv:2504.14642,

    arXiv

  17. [23]

    Visual instruction tuning.ArXiv, abs/2304.08485,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.ArXiv, abs/2304.08485,

    Pith/arXiv arXiv

  18. [24]

    Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.ArXiv, abs/2503.06520, 2025a

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.ArXiv, abs/2503.06520, 2025a. Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiao wen Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.ArXiv, abs/2503.01785, 2...

    Pith/arXiv arXiv

  19. [25]

    Spartqa:: A textual question answering benchmark for spatial reasoning.arXiv preprint arXiv:2104.05832,

    Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. Spartqa:: A textual question answering benchmark for spatial reasoning.arXiv preprint arXiv:2104.05832,

    Pith/arXiv arXiv

  20. [27]

    Spare: Enhancing spatial reasoning in vision-language models with synthetic data.arXiv preprint arXiv:2504.20648,

    Michael Ogezi and Freda Shi. Spare: Enhancing spatial reasoning in vision-language models with synthetic data.arXiv preprint arXiv:2504.20648,

    Pith/arXiv arXiv

  21. [28]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.ArXiv, abs/2503.07536,

    Yi Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.ArXiv, abs/2503.07536,

    Pith/arXiv arXiv

  22. [29]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824,

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824,

    Pith/arXiv arXiv

  23. [30]

    Tuning computer vision models with task rewards.ArXiv, abs/2302.08242,

    André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards.ArXiv, abs/2302.08242,

    Pith/arXiv arXiv

  24. [31]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y . K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.ArXiv, abs/2402.03300,

    Pith/arXiv arXiv

  25. [32]

    Satori-r1: Incentivizing multimodal reasoning with spatial grounding and verifiable rewards.ArXiv, abs/2505.19094, 2025a

    Chuming Shen, Wei Wei, Xiaoye Qu, and Yu Cheng. Satori-r1: Incentivizing multimodal reasoning with spatial grounding and verifiable rewards.ArXiv, abs/2505.19094, 2025a. Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-r1: A stable and gene...

    arXiv

  26. [33]

    Lego-puzzles: How good are mllms at multi-step spatial reasoning? ArXiv, abs/2503.19990,

    Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. Lego-puzzles: How good are mllms at multi-step spatial reasoning? ArXiv, abs/2503.19990,

    Pith/arXiv arXiv

  27. [34]

    Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Deepali Jain, Ryan C

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montse Gonza- lez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauzá, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunkumar Byra- van, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Os car Chang, José Enrique...

    Pith/arXiv arXiv

  28. [35]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis L Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. ArXiv, abs/2406.16860, 2024a. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yan...

    Pith/arXiv arXiv

  29. [36]

    Learning 3d semantic scene graphs from 3d indoor reconstructions.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3960–3969,

    2020

  30. [37]

    Svqa-r1: Reinforcing spatial reasoning in mllms via view-consistent reward optimization.ArXiv, abs/2506.01371,

    Peiyao Wang and Haibin Ling. Svqa-r1: Reinforcing spatial reasoning in mllms via view-consistent reward optimization.ArXiv, abs/2506.01371,

    Pith/arXiv arXiv

  31. [38]

    Compo- sitional 4d dynamic scenes understanding with physics priors for video question answering.arXiv preprint arXiv:2406.00622,

    Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Compo- sitional 4d dynamic scenes understanding with physics priors for video question answering.arXiv preprint arXiv:2406.00622,

    Pith/arXiv arXiv

  32. [39]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models.ArXiv, abs/2201.11903,

    Pith/arXiv arXiv

  33. [40]

    V*: Guided visual search as a core mechanism in multimodal llms.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    16 Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13084–13094,

    2024

  34. [41]

    Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou

    Jiaer Xia, Y .-F. Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-r1: Mitigating shortcuts in visual reasoning with reinforcement learning.ArXiv, abs/2505.14677,

    arXiv

  35. [42]

    Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward.ArXiv, abs/2506.07218,

    Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multimodal reasoning capabilities of multimodal large language models via visual perception reward.ArXiv, abs/2506.07218,

    arXiv

  36. [43]

    Evaluating spatial understanding of large language models.arXiv preprint arXiv:2310.14540,

    Yutaro Yamada, Yihan Bao, Andrew K Lampinen, Jungo Kasai, and Ilker Yildirim. Evaluating spatial understanding of large language models.arXiv preprint arXiv:2310.14540,

    Pith/arXiv arXiv

  37. [45]

    Dapo: An open-source llm reinforcement learning system at scale.ArXiv, abs/2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan Dai, Yuxuan Song, Xiang Wei, Haodong Zhou, Jingjing Liu, ...

    Pith/arXiv arXiv

  38. [46]

    Robopoint: A vision-language model for spatial affordance prediction for robotics.ArXiv, abs/2406.10721,

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.ArXiv, abs/2406.10721,

    Pith/arXiv arXiv

  39. [47]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?ArXiv, abs/2408.13257,

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Jun Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tien-Ping Tan. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?ArXiv, abs/2408.13257,

    Pith/arXiv arXiv

  40. [48]

    Lla- mafactory: Unified efficient fine-tuning of 100+ language models.ArXiv, abs/2403.13372,

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Lla- mafactory: Unified efficient fine-tuning of 100+ language models.ArXiv, abs/2403.13372,

    Pith/arXiv arXiv

  41. [49]

    Easyr1: An efficient, scalable, multi-modality rl training framework.arXiv preprint arXiv:2501.12345,

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework.arXiv preprint arXiv:2501.12345,

    Pith/arXiv arXiv

  42. [50]

    aha moment

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s "aha moment" in visual reasoning on a 2b non-sft model.ArXiv, abs/2503.05132,

    Pith/arXiv arXiv

  43. [51]

    Struct2d: A perception-guided framework for spatial reasoning in large multimodal models.ArXiv, abs/2506.04220,

    Fangrui Zhu, Hanhui Wang, Yiming Xie, Jing Gu, Tianye Ding, Jianwei Yang, and Huaizu Jiang. Struct2d: A perception-guided framework for spatial reasoning in large multimodal models.ArXiv, abs/2506.04220,

    arXiv

  44. [52]

    provides dense, human-annotated scene graphs that support strict grounding of both question gener- ation and answer verification within a unified representational framework. Using Visual Genome, we synthetically constructed a spatial visual question answering dataset called SPATIALTHINKER Visual Question Answering dataset i.e., STVQA-7K comprising 7,587 s...

    2017

  45. [53]

    as an external validation model. For each generated question-answer pair, we assess agreement between the external model and 18 Figure 5: Examples of generated QA pairs across the nine spatial reasoning categories in STVQA-7K. Each category highlights distinct reasoning skills, ranging from relative spatial relations and depth ordering to distance, size, ...

    2025

  46. [56]

    These serve as upper bounds for spatial generalization under non-public training regimes

    and Claude 3.5 Sonnet (CLAUDE-3.5-SONNET-0620) (Anthropic, 2024), which represent the current state-of-the-art in commercial multimodal reasoning. These serve as upper bounds for spatial generalization under non-public training regimes. Open-Source Generalist MLLM Baselines.We compare against generalist open -source MLLMs including Qwen2.5-VL 3B and 7B mo...

    2024

  47. [57]

    which instruction-tunes an MLLM to predict image key-point affordances for robotics and spatial affordance tasks, SpaceThinker (AI & Mayorquin, 2025c), a fine-tuned VLAA-Thinker model for spatial reasoning, and its improved successor SpaceOm (AI & Mayorquin, 2025b), which incorporates deeper chain-of-thought traces and Robo2VLM data (Chen et al., 2025b). ...

    2024

  48. [59]

    with Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. The training configuration employs LoRA with rank 8 applied to all available modules within the model architecture, enabling comprehensive adaptation while maintaining computational efficiency. Models are trained for 3 epochs totaling 645 training steps, using a context window length of ...

    2048

  49. [60]

    aha moment

    as the advantage estimation method, configured with a rollout size of 8 samples per query at a sampling temperature of 1.0. This configuration balances exploration diversity with computational efficiency, allowing the model to discover multiple reasoning strategies while maintaining stable convergence. The training process utilizes a rollout batch size of...

    2048

  50. [61]

    In contrast, Huang et al

    argue that KL regularization can unnecessarily constrain policy updates and recommend removing the KL penalty entirely to allow freer exploration. In contrast, Huang et al. (2024) revisit divergence regularization and propose using a chi-squared penalty to better control overoptimization. Motivated by these findings, we ablate the effect of different dive...

    2024

  51. [128]

    Training time totals around 13 hours for the 3B model and 15 hours for the 7B model

    We train for 75 training steps i.e., 5 training episodes) on 4 × NVIDIA H100 80GB GPUs. Training time totals around 13 hours for the 3B model and 15 hours for the 7B model. The models are trained on high-resolution image inputs ranging from512×512 to 2048×2048 pixels, to preserve fine-grained spatial information. All model parameters, including the vision...

    2048

  52. [2016]

    Computational modeling

    ISSN 2352-1546. Computational modeling. Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.ArXiv, abs/2503.10615, 2025b. Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bo...

    Pith/arXiv arXiv

  53. [2017]

    Topviewrs: Vision-language models as top-view spatial reasoners.arXiv preprint arXiv:2406.02537, 2024a

    Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, and Ivan Vuli ´c. Topviewrs: Vision-language models as top-view spatial reasoners.arXiv preprint arXiv:2406.02537, 2024a. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal ...

    Pith/arXiv arXiv

  54. [2019]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  55. [2020]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465, 2024a. Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role-p...

    Pith/arXiv arXiv

  56. [2021]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms.ArXiv, abs/2402.07872,

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Ho Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang- Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Manfred Otto Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. Pivot: Iterative visual pro...

    Pith/arXiv arXiv

  57. [2022]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaoshen Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. ArXiv, abs/2503.06749,

    Pith/arXiv arXiv

  58. [2023]

    Compile scene graphs with reinforcement learning.arXiv preprint arXiv:2504.13617, 2025c

    11 Zuyao Chen, Jinlin Wu, Zhen Lei, Marc Pollefeys, and Chang Wen Chen. Compile scene graphs with reinforcement learning.arXiv preprint arXiv:2504.13617, 2025c. An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models.Advances in Neural Informa...

    Pith/arXiv arXiv

  59. [2024]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, nathan lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.ArXiv, abs/2503.01307,

    Pith/arXiv arXiv

  60. [2025]

    Qwen2.5-vl technical report.ArXiv, abs/2502.13923,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.Ar...

    Pith/arXiv arXiv

  61. [2048]

    For models with spe- cific reasoning templates such as VLAA-Thinker, SpaceThinker, and SpaceOm, we utilize their corresponding structured prompts

    to ensure deterministic and reproducible outputs. For models with spe- cific reasoning templates such as VLAA-Thinker, SpaceThinker, and SpaceOm, we utilize their corresponding structured prompts. In line with their original training setup, SpatialRGPT receives depth inputs, while all other models are evaluated using RGB images alone. Our evaluation pipel...

    2025