Pith. sign in

REVIEW 4 major objections 5 minor 10 cited by

ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ACTIVE-O3 uses GRPO reinforcement learning to teach a multimodal language model where to zoom in an image, improving small-object detection and interactive segmentation under a fixed sensing budget.

desk verdict A useful framework and benchmark for MLLM active perception, but the central claim that RL learns 'where to look' is not yet supported by the experiments. read the letter →

arxiv 2505.21457 v2 pith:VNAKPWUD submitted 2025-05-27 cs.CV cs.AI

Muzhi Zhu , Hao Zhong , Canyu Zhao , Zongze Du , Mingyu Liu , Zheng Huang , Anzhou Li , Hao Chen
show 4 more authors
Cheng Zou Jingdong Chen Ming Yang Chunhua Shen
This is my paper · ORCID
classification cs.CVcs.AI
keywords activeperceptionvisionmultimodallargelanguagemodelsGRPOreinforcementlearningregionproposalsmallobjectdetectioninteractivesegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ACTIVE-O3 asks whether a multimodal large language model can learn active perception—deciding where to zoom in an image—using only reinforcement learning, without any labels for good regions. The paper argues yes: with Group Relative Policy Optimization (GRPO), a lightweight policy-gradient update that needs no separate critic model, and a reward combining task success with simple heuristics about format, overlap, and region size, a single model learns to propose informative crops. These crops are then fed to a task model, and the whole loop improves small-object grounding on LVIS, small-object detection on SODA-A and SODA-D, and interactive segmentation on ThinObjects under a fixed sensing budget. The authors also show the trained sensing policy transfers zero-shot to fine-grained visual question answering such as the V* benchmark, where it outperforms a chain-of-thought baseline.

What carries the argument

The load-bearing mechanism is the modular sensing–action decomposition combined with Group Relative Policy Optimization (GRPO), a policy-gradient update that estimates advantages from the mean and standard deviation of rewards within a group of sampled responses and therefore needs no critic model. The sensing policy $M_O$ is the same multimodal language model prompted to output up to three bounding boxes; each crop is resized and passed to the task model $M_A$, and the reward is a weighted sum of task-aware terms ($AP+AR$ for detection, $mIoU$ after SAM-based interactive segmentation) and heuristic terms (JSON validity, non-overlap threshold $\tau=0.3$, area ratio between 0.01 and 0.5, and coverage of ground-truth masks or boxes). Under the fixed-budget objective $\max_{M_O} \mathbb{E}\left[\sum_{k=1}^K R(M_A(o_k), \mathcal{I})\right]$, the reward signal is what transfers downstream utility back to region selection.

What would settle it

Train ACTIVE-O3 under the same rewards and data but with a fixed or random crop policy instead of the learned sensing policy, and compare downstream $AP$/$AR$ and $mIoU$; if that control matches or beats ACTIVE-O3, the claim that region selection is learned would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that MLLM-based active perception can be formalized as a two-stage policy—a sensing policy $M_O$ proposing up to $K$ rectangular regions from a low-resolution global view, and a task model $M_A$ operating on the crops—and that GRPO, with a dual-form reward, can learn the sensing policy directly from downstream task feedback. The paper demonstrates the claim by training Qwen2.5-VL-7B-Instruct this way and reporting consistent improvements: on LVIS small and dense grounding, $AP_s$ and $AR_s$ rise by +1.0/+2.8 and +2.7/+3.5 over Qwen2.5-VL-CoT; on SODA-A and SODA-D, $AP_s$ improves by +8.5 and +13.0; and on ThinObjects interactive segmentation, $mIoU$ at zoom budget 3 rises from 0.796 to 0.863 while the chain-of-thought baseline falls to 0.561. The paper further claims the learned sensing policy is reusable: paired with Grounding DINO it improves that detector, and it transfers zero-shot to fine-grained VQA without being trained on reasoning or question-answering data.

Load-bearing premise

The load-bearing premise is that the task model can be treated as fixed while only the sensing policy is learned; in the implementation both roles are played by the same Qwen2.5-VL-7B model updated by GRPO, so gains could in principle come from improving the shared model on detection and segmentation rather than from learning where to look.

Editorial extensions

If this is right

  • A model trained only on object-level rewards learns to emit spatial proposals, suggesting region-selection skill can emerge without region annotations.
  • With a fixed sensing budget of three regions, ACTIVE-O3 consistently beats the chain-of-thought prompting baseline on $AP_s$/$AR_s$ for small and dense objects and on $mIoU$ for interactive segmentation.
  • The same sensing policy can be paired at test time with a different, stronger task model such as Grounding DINO, improving its results and supporting the modularity of the design.
  • The policy transfers zero-shot to fine-grained VQA such as V*, so detection-oriented active perception training appears to generalize to other tasks that require zooming.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be a control in which crops come from a fixed or random policy under identical rewards; if that control matched ACTIVE-O3, the gains would come from the shared backbone improving on detection and segmentation rather than from learned region selection.
  • The modular formulation suggests a direct path to embodied active perception: replace the crop action with camera-viewpoint control and keep the same reward structure, which the static-image experiments here do not test.
  • Measuring accuracy against the number of crops used, rather than only against a fixed budget, would connect this work to compute-adaptive inference, where the model spends resolution only where the task demands.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper formalizes MLLM-based active perception as a two-module policy—a sensing model M_O that proposes regions and a task model M_A that executes on crops—and proposes ACTIVE-O3, a GRPO-based reinforcement learning framework with a dual-form reward (heuristic plus task-aware). Experiments are reported on LVIS small/dense grounding, SODA-A/SODA-D small-object detection, and ThinObjects interactive segmentation, with qualitative zero-shot examples on the V* benchmark. The central claim is that ACTIVE-O3 learns efficient, stable region-selection strategies without explicit region-selection supervision, substantially improving active perception over Qwen2.5-VL-CoT under a fixed sensing budget.

Significance. If the attribution of gains to the learned sensing policy were established, ACTIVE-O3 would be a practical contribution to MLLM-based active vision, leveraging pure RL without separate critic or SFT. The paper provides a clean task definition, a modular formulation, a released codebase, and a multi-domain benchmark, which are strengths. However, the current experiments do not exclude an alternative explanation—that improvements come from fine-tuning the shared task/sensing backbone on detection and segmentation rewards—and the claimed absence of explicit region-selection supervision is contradicted by a ground-truth-based coverage reward. The significance of the specific active-perception claim is therefore not yet supported.

major comments (4)
  1. [Section 4.2, Eq. (2), Appendix E.2, Remark D.2] The formal objective in Eq. (1) assumes M_A is fixed and optimizes only M_O, but the implementation instantiates both M_A and M_O with the same Qwen2.5-VL-7B-Instruct model and updates it with the GRPO objective in Eq. (2). As a result, the gains in Tables 1 and 2 could come from improving the shared backbone on detection rewards rather than from learning where to look. The decoupling experiment ACTIVE-O3+GDINO in Table 1 replaces M_A with Grounding DINO at test time, but it is not compared against GDINO using CoT-proposed regions, so it does not isolate the sensing policy. Please add a control that trains with fixed or random crops under the same reward, or freezes M_A during RL training.
  2. [Abstract and Section 1] The abstract promises that ACTIVE-O3 'can also serve as a proxy task for leveraging perception data, further improving performance on benchmarks such as RealWorldQA and MME,' but the manuscript contains no evaluation on RealWorldQA or MME. Either add the missing results or remove this claim from the abstract, since it is presently unsupported.
  3. [Appendix B.4 and Section 1] The abstract and Section 1 claim that ACTIVE-O3 learns region selection 'without explicit region-selection supervision,' but the coverage-based reward R_coverage in Appendix B.4 directly uses ground-truth masks and bounding boxes with thresholds θ and δ. This is explicit supervision of the selected regions, and the ablation in Table 4 shows that the combined reward includes this component. Please revise the claim or provide an ablation without R_coverage to show that the method works without it.
  4. [Appendix E.3 and Table 2] The SODA evaluation lowers the IoU threshold to 0.1 for detection metrics, which is non-standard and likely inflates the AP/AR values reported in Table 2. The paper should also report results under the standard COCO IoU range (0.5:0.95) or provide a systematic justification for the threshold and a sensitivity analysis, so that the advertised gains are comparable with prior work.
minor comments (5)
  1. [Figure 6] The word 'signicade' appears to be a typo; the intended word is likely 'sign' or 'barricade.'
  2. [Section 4] The notation M_A and M_O is used before the mapping to 'task model' and 'sensing model' is stated; a sentence clarifying the correspondence immediately after Eq. (1) would improve readability.
  3. [Section 1 and Appendix D.4] The paper repeatedly refers to 'GPT-o3' without a citation; if this is a real model, a reference is needed, and if it is a hypothetical or renamed model, the text should say so.
  4. [Appendix G.1] The claim of zero-shot transfer to the V* benchmark is supported only by qualitative examples in Figures 1, 10–12; a quantitative table on V* would make the claim verifiable.
  5. [Table 3] The row label 'LVIS + SODA-A + D' is ambiguous; it should be written as 'LVIS + SODA-A + SODA-D' for clarity.

Circularity Check

2 steps flagged · score 4.0 of 10

Coverage reward computed from GT boxes/masks contradicts the 'no explicit region-selection supervision' claim, and the shared-weight implementation breaks the fixed-MA assumption that would isolate the sensing policy.

  1. self definitional [Abstract; Appendix B.4 (Coverage-Based Reward) and B.5 (Overall Heuristic Reward)]
    "Coverage-Based Reward. When ground truth masks or boxes are available, we assess how well the predicted regions align with task-relevant areas. This can include: (i) the proportion of ground-truth mask pixels covered by a region, (ii) the percentage of ground-truth boxes matched by at least one proposal."

    The abstract's central claim is that ACTIVE-O3 'autonomously learns efficient and stable region selection strategies without explicit region-selection supervision.' But R_heuristic (B.5) includes λ4 R_coverage, and R_coverage is defined directly from GT masks/boxes: a proposal is rewarded for covering GT pixels or matching GT boxes. The sensing policy is therefore optimized against region-level GT, so the claimed absence of explicit region-selection supervision is contradicted by the paper's own reward construction. The 'autonomous' region-selection result is not derived; it is built into the reward by definition.

  2. other [Section 3 Eq. (1); Section 4; Appendix E.2; Remark D.2]
    "In the 2D setting, we assume a fixed task model MA and focus on learning a sensing policy MO that selects K informative regions from a static image I based on an initial observation oinit and instruction I."

    Eq. (1) optimizes only MO with MA fixed, but Appendix E.2 states 'We use Qwen2.5-VL-7B-Instruct as the shared policy backbone πθ,' and Section 4 instantiates both MA and MO as parses of the same π. Thus the 'fixed' task model is co-trained during GRPO. Observed LVIS/SODA gains may come from improving the shared model on detection rewards rather than from learning where to look, and the ACTIVE-O3+GDINO decoupling is not compared against GDINO with CoT-proposed regions. This is an attribution confound rather than an equation-level identity, but it prevents the central sensing-policy claim from being established by the stated optimization.

full rationale

The paper contains no load-bearing self-citation or imported uniqueness theorem, and the held-out LVIS/SODA evaluations provide independent evidence that the training recipe improves downstream detection. However, two load-bearing steps in the claimed chain do not hold as stated. First, the 'without explicit region-selection supervision' contribution is contradicted by the GT-based coverage reward in Appendix B.4; the learned region proposals are rewarded for matching GT boxes/masks, so this part of the claim reduces to the reward definition. Second, the formal objective assumes MA is fixed while the implementation shares and updates one MLLM for both MA and MO, so the measured gains are not cleanly attributable to the sensing policy. These issues make the central narrative partially circular/overclaimed, but they do not collapse the empirical comparisons, so a moderate score is appropriate.

Assumptions & free parameters 6 free parameters · 6 assumptions · 1 invented entities

The framework depends on several hand-set reward parameters and benchmark-specific evaluation choices, on a fixed-task-model assumption that the implementation violates, and on ground-truth-based rewards despite the claim of no explicit region-selection supervision. No new physical entities are introduced; the only invented component is the oracle segmentation evaluator.

free parameters (6)
  • Heuristic reward weights lambda_1..lambda_4 = all set to 1
    Appendix B.5 sets equal weights without sensitivity analysis; the balance between format, overlap, area, and coverage rewards is hand-chosen.
  • Non-overlap IoU threshold tau = 0.3
    Appendix B.2; proposals with any pairwise IoU above 0.3 get zero non-overlap reward.
  • Area ratio bounds = 0.01 to 0.5 of image area
    Appendix B.3; hand-set acceptable box size range.
  • Coverage and matching thresholds theta, delta = not specified
    Appendix B.4 defines Coverage(b_i, M) >= theta and IoU >= delta but does not give values, so the reward is not fully reproducible.
  • SODA evaluation IoU threshold = 0.1
    Appendix E.3 lowers the COCO-style IoU threshold to 0.1 to make domain scores comparable; a post hoc evaluation choice.
  • Number of proposed regions K = 3
    All experiments use up to three zoom-in regions; fixed sensing budget rather than learned.
assumptions (6)
  • standard math GRPO with clipped importance ratios and KL regularization is a valid optimizer for the MLLM policy (Eq. 2).
    Taken from DeepSeek-R1 [17]; the paper does not re-derive it.
  • domain assumption The task model M_A can be treated as fixed while only the sensing policy M_O is optimized.
    Section 4.2 and Remark D.2 assume a reasonably strong fixed M_A, but Appendix E.2 instantiates both M_A and M_O with Qwen2.5-VL-7B-Instruct, so the assumption is violated.
  • domain assumption Rewards computed from ground-truth boxes and masks are a valid training signal for the sensing policy.
    Appendix B.4 and C use ground-truth masks and boxes; this is standard supervised signal but contradicts the claim of no explicit region-selection supervision.
  • domain assumption Static 2D crop selection with up to three boxes is a representative proxy for active perception.
    Section 3 specializes embodied active perception to static images and a fixed budget; the authors acknowledge the simplification in Remark D.3.
  • ad hoc to paper GPT-o3 uses a zoom-in search strategy and is inefficient and inaccurate.
    Introduction and Remark D.4 describe GPT-o3 behavior without citation or quantitative comparison; the paper's motivation partly rests on this assertion.
  • domain assumption Training on 1,000 images from each target-domain SODA split supports claims of domain generalization.
    Appendix E.3; the model sees target-domain training data, so domain generalization is within-domain transfer, not zero-shot domain shift.
invented entities (1)
  • Oracle task model for interactive segmentation
    purpose: Simulates perfect click-based feedback so the segmentation reward (SAM plus oracle clicks) isolates the sensing policy; it is not a real deployable task model.
    Appendix E.3 and Section 5.4; without a public strong M_A the authors construct an oracle, so segmentation numbers are not end-to-end system results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning." pith.science (2026). https://pith.science/paper/VNAKPWUD

@misc{pith2026250521457,
  author       = {Pith},
  title        = {Pith review of: ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNAKPWUD}},
  note         = {Machine review of arXiv:2505.21457}
}
read the original abstract

Active vision, also known as active perception, refers to actively selecting where and how to look in order to gather task-relevant information. It is a critical component of efficient perception and decision-making in humans and advanced embodied agents. With the rise of Multimodal Large Language Models (MLLMs) as central planners in robotic systems, the lack of methods for equipping MLLMs with active perception has become a key gap. We first provide a systematic definition of MLLM-based active perception tasks and show that GPT-o3's zoom-in strategy can be viewed as a special case, though it suffers from low efficiency and inaccurate region selection. To address these issues, we propose ACTIVE-o3, a reinforcement learning framework built on GRPO that equips MLLMs with active perception capabilities. Leveraging a modular sensing-action design and a dual-form reward, ACTIVE-o3 autonomously learns efficient and stable region selection strategies without explicit region-selection supervision. We further establish a comprehensive benchmark covering both open-world tasks, including small- and dense-object grounding, and domain-specific scenarios, including remote sensing, autonomous driving, and interactive segmentation. Experimental results demonstrate that ACTIVE-o3 significantly enhances active perception capabilities compared to baselines. Moreover, we show that our framework not only preserves the model's general understanding ability but can also serve as a proxy task for leveraging perception data, further improving performance on benchmarks such as RealWorldQA and MME.

Figures

Figures reproduced from arXiv: 2505.21457 by the authors.

Figure 1
Figure 1. Zero-shot reasoning on the V ∗ benchmark. When asked “Tell me the number on the traffic light”, Qwen2.5 VL incorrectly refers to unrelated text. In contrast, ACTIVE-O3 locates and magnifies the precise area on the traffic light, accurately answering 10 through effective spatial localization. With the recent surge in the capabilities of multimodal large language models (MLLMs) [7, 8, 9, 10], these models are increasi… view at source ↗
Figure 2
Figure 2. Overview of the proposed Active-O3 framework. Given a multimodal query (e.g., “find [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prompt for ACTIVE-O3-DET. Specialization to 2D Visual Scenes While our general formulation applies to embodied agents in complex physical environments, such settings are often difficult to deploy and evaluate in a reproducible manner. To facilitate more controlled and fair comparisons, we specialize the problem to a simplified yet representative 2D scenario: active perception over static images. In this setting, the… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Visualization details of our proposed method on three datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of segmentation performance (mIoU) under different zoom-in budgets. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: A failure case of GPT-o3 in answering the question: What animal is drawn on that red [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt for ACTIVE-O3-DET. Prompt for ACTIVE-O3 Segmentation • "Identify exactly three distinct regions in the image that illustrate segmentation inaccuracies in the translucent green mask for the ‘{object}’." • "The selected regions should be as distinct as possible, w…
Figure 8
Figure 8. Figure 8: Prompt for ACTIVE-O3-Seg. storage tanks), which can lead to inaccurate task-aware reward estimation due to the limited capability of the task model. Second, the current action space is constrained. Our framework only allows zooming into three target regions per step. H…
Figure 9
Figure 9. Figure 9: Prompt for the task model MA. E.2 Implementation Details We use Qwen2.5-VL-7B-Instruct as the shared policy backbone πθ. All experiments are conducted using GRPO with KL regularization coefficient β = 0.04, group size 8, and a learning rate of 1e−6 using the AdamW opti…
Figure 10
Figure 10. Figure 10: Zero-shot reasoning on the V ∗ benchmark (Example 2). Given the question “Tell me the number on the police car”, the baseline model (Qwen2.5 VL) fails to locate the relevant visual evidence due to limited resolution and reasoning capability. In contrast, our method (A…
Figure 11
Figure 11. Figure 11: Zero-shot reasoning on the V ∗ benchmark (Example 3). For the question “What is the color of the van?”, the baseline model (Qwen2.5 VL) fails to detect the presence of the van and incorrectly claims that no such object is visible. In contrast, ACTIVE-O3 accurately ide…
Figure 12
Figure 12. Figure 12: Zero-shot reasoning on the V ∗ benchmark (Example 4). Given the question “What is the color of the watchband?”, baseline predictions are inconsistent. ACTIVE-O3 focuses on the wrist of the foreground figure, providing the accurate answer (purple) by effectively zoomin…
Figure 13
Figure 13. Figure 13: Visualization of Small Object Detection results on SODA-A and SODA-D datasets. Each row shows a different [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Visualization of object detection results on various scenes from the LVIS dataset. The left column shows [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Interactive segmentation analysis on ThinObjects. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Failure cases. Left (LVIS): When objects are densely packed, the model fails to distinguish between them, resulting in inaccurate segmentation. Right (SODA-A): For small objects in aerial images, domain gap issues lead to poor localization—even if the object is roughl…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

    cs.CV 2026-05 accept novelty 8.0 of 10

    Introduces the TVR active viewpoint-matching task and TVRBench indoor simulation benchmark, where foundation models start at low single-digit success rates but reach 51.4% after visual-action SFT and multi-turn GRPO p...

  2. Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Authors create ReasonMatch-Bench and DCRL training to boost MLLM performance on wide-baseline matching, reporting gains over baselines while preserving general capabilities.

  3. PInVerify: An Offline Embodied Benchmark for Active Instance Verification

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    PInVerify is a new offline embodied benchmark for active instance verification that supplies multi-view captures and 6-sector navigation topology, with MLLM baselines reaching 85.6% after fine-tuning but showing no re...

  4. Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Visual CoT agents exhibit tool-use collapse where tool usage declines but task accuracy rises, and adding entropy regularization for rollout diversity produces the strongest performance.

  5. InSight-doc: Agentic Visual Perception for Long-Document Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    InSight-doc trains an 8B vision-language model to zoom into document sub-regions on demand, improving long-document VQA accuracy by up to 16.4 points while cutting latency by 41-68%.

  6. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  7. EAGLE-360: Embodied Active Global-to-Local Exploration in 360$^\circ$

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    EAGLE-360 introduces a global-to-local exploration framework for 360° visual search, adapting RoPE Rolling, creating a new VQA dataset, and using SFT+GRPO training to claim SOTA performance with 8x accuracy gain.

  8. Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    LASER uses self-evolving preference optimization to teach GUI models to crop and zoom before clicking, reaching 55.7% on ScreenSpot-Pro with a 7B model.

  9. From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    The survey formalizes MLLM perception as a unified vision-language capability and traces its evolution via a new five-stage taxonomy while outlining future challenges.

  10. Latent Visual States for Efficient Multimodal Reasoning

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    EVA generates adaptive Latent_slot tokens as internal visual thoughts, trained end-to-end with text tokens via D-GSPO on the EVA-230K dataset, claiming performance gains and better inference efficiency.

Reference graph

Works this paper leans on

44 extracted references · 18 canonical work pages · cited by 10 Pith papers

  1. [1]

    Active vision.International journal of computer vision, 1:333–356, 1988

    John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision.International journal of computer vision, 1:333–356, 1988. 1, 3

    work page 1988

  2. [2]

    Animate vision.Artificial intelligence, 48(1):57–86, 1991

    Dana H Ballard. Animate vision.Artificial intelligence, 48(1):57–86, 1991. 1, 3

    work page 1991

  3. [3]

    Autonomous exploration: Driven by uncertainty.IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(3):193–205, 1997

    Peter Whaite and Frank P Ferrie. Autonomous exploration: Driven by uncertainty.IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(3):193–205, 1997. 1, 3

    work page 1997

  4. [4]

    Active vision for dexterous grasping of novel objects

    Ermano Arruda, Jeremy Wyatt, and Marek Kopicki. Active vision for dexterous grasping of novel objects. In2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2881–2888. IEEE, 2016. 1, 4

    work page 2016

  5. [5]

    Neural modular control for embodied question answering

    Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. InConference on robot learning, pages 53–62. PMLR, 2018. 1, 4

    work page 2018

  6. [6]

    Learning to explore using active neural slam.arXiv preprint arXiv:2004.05155, 2020

    Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhut- dinov. Learning to explore using active neural slam.arXiv preprint arXiv:2004.05155, 2020. 1, 3, 4

    arXiv 2004

  7. [7]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 2, 3

    arXiv 2023

  8. [8]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 2

    arXiv 2024

Show all 44 references
  1. [9]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. 2, 3

    2024 arXiv

  2. [10]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

    arXiv

  3. [11]

    Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025

    Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, et al. Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation.arXiv preprint arXiv:2502.13143, 2025. 2

    2025

  4. [12]

    Magma: A foundation model for multimodal ai agents

    Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. arXiv preprint arXiv:2502.13130, 2025. 2

    2025 arXiv

  5. [13]

    Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020, 2025

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:25...

    2025 arXiv

  6. [14]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 2, 4

    2024 arXiv

  7. [15]

    pi0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024. 2, 4

    arXiv

  8. [16]

    \pi_0.5: A vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. \pi_0.5: A vision- language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025. 2

    2025 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 3

    2025 arXiv

  10. [18]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024. 3, 17 25

    2024

  11. [19]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  12. [20]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 3

    2023 arXiv

  13. [21]

    Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024. 3

    2024 arXiv

  14. [22]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 3

    2023

  15. [23]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  16. [24]

    Deepseek-vl: towards real-world vision-language under- standing.arXiv preprint arXiv:2403.05525, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language under- standing.arXiv preprint arXiv:2403.05525, 2024. 3

    2024 arXiv

  17. [25]

    Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302,

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302,

    arXiv

  18. [26]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  19. [27]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 3

    2024 arXiv

  20. [28]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning, 2025

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning, 2025. 3

    2025

  21. [29]

    Video-r1: Reinforcing video reasoning in mllms, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms, 2025. 3

    2025

  22. [30]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785,

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785,

    arXiv

  23. [31]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025. 3

    2025 arXiv

  24. [32]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025. 3

    2025 arXiv

  25. [33]

    Iqa: Visual question answering in interactive environments

    Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4089–4098, 2018. 3

    2018

  26. [34]

    Object goal navigation using goal-oriented semantic exploration.Advances in Neural Information Processing Systems, 33:4247–4258, 2020

    Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhut- dinov. Object goal navigation using goal-oriented semantic exploration.Advances in Neural Information Processing Systems, 33:4247–4258, 2020. 3

    2020

  27. [35]

    Learning to look around: Intelligently exploring unseen environments for unknown tasks

    Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1238–1247, 2018. 3, 4 26

    2018

  28. [36]

    Target-driven visual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IEEE, 2017. 3

    2017

  29. [37]

    Objectnav revisited: On evaluation of embodied agents navigating to objects.arXiv preprint arXiv:2006.13171, 2020

    Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects.arXiv preprint arXiv:2006.13171, 2020. 3

    2006 arXiv

  30. [38]

    Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation, 2025

    Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, and Li Yi. Sofar: Language-grounded orientation bridges spatial reasoning...

    2025

  31. [39]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer,

  32. [40]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024. 9

    2024 arXiv

  33. [41]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 9, 16

    2019

  34. [42]

    Towards large-scale small object detection: Survey and benchmarks.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13467–13488, 2023

    Gong Cheng, Xiang Yuan, Xiwen Yao, Kebing Yan, Qinghua Zeng, Xingxing Xie, and Junwei Han. Towards large-scale small object detection: Survey and benchmarks.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13467–13488, 2023. 10, 16

    2023

  35. [43]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 13

    2024 arXiv

  36. [44]

    Deep interactive thin object selection

    Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 305–314, 2021. 16 27

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.