Pith. sign in

REVIEW 3 major objections 5 minor 80 references

A multimodal model can jointly train region captioning and localization with only masks, by scoring its own captions through how well it re-localizes them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:36 UTC pith:GXNAT3E7

load-bearing objection Clean dual-task RL loop that actually removes the text-mask annotation bottleneck and shows multi-benchmark gains; the reward-faithfulness assumption is the soft spot but the ablations already check it. the 3 major comments →

arxiv 2607.11581 v1 pith:GXNAT3E7 submitted 2026-07-13 cs.CV

Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

Xin Zhang , Haochen Wang , Yikang Zhou , Jason Li , Robby T. Tan This is my paper
classification cs.CV
keywords Multimodal Large Language ModelsReinforcement LearningRegion UnderstandingReferring LocalizationCycle ConsistencyCycleGRPOPixel-level MLLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Region understanding and referring localization are dual problems: describe a given region, or find the region from a description. This paper argues that one multimodal model can improve at both without any textual ground truths by closing the loop region to text to region. The model first generates candidate captions for an input mask or box, then immediately tries to ground those captions back into space; reconstructive accuracy becomes the reward for both directions. Built as CycleGRPO on a mask-token backbone and trained only on image–region pairs, the method lifts captioning, region VQA, grounded dialogue, and referring segmentation without task-specific fine-tuning. A sympathetic reader cares because fine-grained region–caption pairs are expensive and scarce, while a self-evaluating cycle can scale from region inputs alone.

Core claim

CycleGRPO jointly bootstraps region understanding and localization by maximizing a cycle-consistency reward: a single MLLM acts as actor to sample captions for a region, then as its own critic to localize those captions back into masks or boxes. Average reconstructive IoU across localization rollouts measures caption quality and weights the grounding trajectories, so the policy is optimized under group-relative policy optimization without text labels, external judges, or separate task heads. Empirically this raises performance across captioning, VQA, interleaved text–mask generation, and referring segmentation in a zero-shot evaluation setting.

What carries the argument

CycleGRPO (“Actor as Its Own Critic”): for each region the policy samples G captions, then K localization rollouts per caption; reconstructive IoU supplies a quality-aware cycle reward (caption reward = mean IoU; localization reward = that mean times per-rollout IoU) that trains both mappings in one GRPO update.

Load-bearing premise

The paper assumes that average reconstructive localization accuracy is a faithful proxy for how discriminative and correct a generated caption is.

What would settle it

After CycleGRPO training, check whether captions with high average reconstructive IoU also win on independent caption quality (human ratings or a held-out judge on attribute correctness and discriminability). If high-IoU captions remain vague or wrong on details that barely change mask overlap, the cycle reward is not measuring semantic quality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pixel-level MLLM post-training can scale from image–mask (or box) data alone instead of costly region–caption corpora.
  • Separate captioning and grounding RL pipelines can be replaced by one closed loop that improves both capabilities together.
  • The same cycle applies to discrete mask tokens and continuous bounding-box formats across model sizes.
  • Zero-shot gains on region captioning, region VQA, grounded dialogue, and referring segmentation follow without per-benchmark fine-tuning.
  • Captions that fail to reconstruct the original region are penalized, which the paper links to fewer hallucinations and stronger target rejection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same duality could extend to video or 3D regions wherever a reconstructible spatial token or box representation exists.
  • Because mask reconstruction rewards attributes more than relations, the cycle may systematically under-train spatial-relation language unless the localization metric is enriched.
  • If localization saturates while captions stay shallow, an auxiliary diversity or coverage term would be a natural next control the paper does not introduce.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CycleGRPO, a GRPO-based RL framework that jointly optimizes region understanding and localization in MLLMs by exploiting their duality as a closed loop “region → text → region.” A single model first samples candidate region captions (actor), then grounds each caption back into spatial masks via K localization rollouts (critic). A cycle-consistency reward—defined in the main text as reconstructive IoU, and in the appendix as hierarchical token grading—supervises both phases without textual ground truths. Built on SAMTok (Qwen3-VL-4B) and trained on ~20k DenseWorld masks plus 1k GRES no-target expressions, the method reports consistent zero-shot gains on DLC-Bench, GAR-Bench-VQA, GCG, GRES, and GroundingSuite, with ablations on per-task GRPO, G/K, bbox vs. mask formats, and data scale, plus code release.

Significance. If the results hold under clean evaluation, the work is a solid systems contribution: it removes the need for expensive region–caption pairs for joint fine-grained post-training and shows that a single self-evaluating loop can improve both captioning and grounding. Strengths include multi-benchmark zero-shot evaluation, explicit Cap-only / Loc-only / CycleGRPO controls (Table 6), transfer to continuous bbox formats on vanilla Qwen backbones (Table 8), G/K and data-scaling ablations, training-dynamics plots, and public code. The “Actor-as-Own-Critic” framing is clear and practically useful for pixel-level MLLM post-training.

major comments (3)
  1. [Sec. 4 Implementation; Table 5] Sec. 4 (Implementation) states training uses “1k no-target expressions from GRES [28]” while Table 5 reports a large GRES N-acc jump (58.7% → 92.1%) as a main result under “no task-specific fine-tuning.” Because N-acc is precisely the no-target rejection metric, this is partial in-domain supervision for the most dramatic GRES number. Please either (i) retrain without any GRES expressions and re-report Table 5, or (ii) clearly mark GRES N-acc as in-domain / semi-supervised and lead with the non-GRES localization gains (GroundingSuite, GCG).
  2. [Sec. 3 Cycle Reward; Appendix A] Sec. 3 defines the cycle score as mask IoU, s_i,k = IoU(M, M̂_i,k), and builds R_cap and R_loc on that quantity. Appendix A instead replaces pixel IoU with hierarchical token grading (exact match 1.0 / prefix 0.8 / partial 0.4) for efficiency. This is not a minor implementation detail: it changes the reward geometry and is the actual training signal. Please (i) state the implemented reward in the main method section, (ii) report a small controlled comparison of true IoU vs. hierarchical token grading on DLC-Bench and GroundingSuite, and (iii) clarify whether reported gains depend on the discrete SAMTok codebook structure.
  3. [Sec. 3 Cycle Reward; Tables 6–8] The load-bearing assumption is that average reconstructive accuracy of K localization rollouts is a faithful, non-degenerate proxy for caption semantic discriminability (Sec. 3, Phase 2 and Cycle Reward). Tables 6–8 and the GRES N-acc behavior provide useful checks against pure collapse, but the paper still lacks a direct analysis of failure modes: e.g., rate of high-reward captions that remain vague or partially hallucinated under human or LLM inspection, and correlation between R_cap and external caption metrics (CIDEr / LLM-judge) on held-out regions. A short quantitative study (even on a few hundred samples) would substantially strengthen the claim that the cycle reward drives genuine understanding rather than tokenizer-friendly phrasing.
minor comments (5)
  1. [Table 3; Sec. 4.1 Region VQA] Table 3: Position (−3.2%) and Relation (−1.0%) drop under CycleGRPO; the text attributes this to attribute-biased reconstruction. Consider a short qualitative example or a relation-focused reward variant discussion so readers can judge the trade-off.
  2. [Fig. 6; Appendix E] Fig. 1 and Fig. 2 are conceptually clear; ensure axis labels / legend in Fig. 6 (training dynamics) remain readable in print and that solid vs. dashed CycleGRPO reward curves are defined in the caption.
  3. [Sec. 3; Eq. (1)] Notation: R_total = R_cap + R_loc is introduced without specifying whether advantages A_cap and A_loc are standardized jointly or per phase; a one-line clarification near Eq. (1) would help reimplementation.
  4. [Sec. 5] Related Work could briefly contrast CapRL and other caption-via-QA RL methods with the pure spatial cycle signal to sharpen novelty.
  5. [Fig. 1; throughout] Minor typos / polish: “SAMToK” vs “SAMTok” inconsistency early on; “unscalablehigh-quality” missing space in Fig. 1 caption; arXiv IDs and venue formatting in references should be checked for consistency.

Circularity Check

0 steps flagged

No circularity: cycle-IoU is an intentional training proxy; all reported gains are on held-out external benchmarks never used in the reward.

full rationale

CycleGRPO is an empirical RL methods paper, not a first-principles derivation. The training objective is deliberately self-referential (R_cap = mean IoU of K localization rollouts of the model's own captions; R_loc weighted by that score; App. A hierarchical token match as a cheap proxy). That design choice does not make the scientific claims circular: every headline number (DLC-Bench Avg, GroundingSuite All, GRES gIoU/N-acc, GCG CIDEr/Recall, GAR-Bench-VQA Overall) is measured on held-out benchmarks that never enter the reward and that receive no task-specific fine-tuning. No free parameter is fitted to a subset of those metrics and then re-reported as a prediction; no uniqueness theorem is imported from overlapping-author work to forbid alternatives; SAMTok is used only as a discrete-mask substrate and is shown to be non-essential by the bbox transfer on vanilla Qwen2.5/3-VL (Tab. 8). Cap-only vs Loc-only vs CycleGRPO (Tab. 6) further shows mutual benefit rather than a tautological collapse. The derivation chain therefore does not reduce any claimed result to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central claim rests on the duality assumption, the discrete-token representation of SAMTok (or bbox tokens), the GRPO update rule, and a handful of sampling and optimization hyper-parameters. No new physical entities are postulated; the invented objects are algorithmic constructs whose only evidence is the reported benchmark gains.

free parameters (3)
  • caption group size G and localization rollout count K = G=K=6
    Set to 6 by default; ablation shows performance rises with G=K but the chosen value is a compute-accuracy trade-off, not derived.
  • learning rate and weight decay = 1e-6 / 1e-2
    AdamW 1e-6 / 1e-2 chosen for the single-epoch run; standard free hyper-parameters.
  • hierarchical token-grading scores (1.0 / 0.8 / 0.4) = 1.0 exact, 0.8 prefix, 0.4 boundary
    Heuristic partial-credit scheme used in the supplement to avoid full mask decoding; values are hand-chosen.
axioms (3)
  • domain assumption Region captioning and referring localization are dual maps whose cycle reconstruction error (IoU) is a valid intrinsic measure of caption discriminability.
    Stated in Sec. 3 (Duality … and L_cycle); underpins the entire reward design.
  • domain assumption SAMTok (or standard bbox tokens) provides a unified discrete vocabulary in which both text and spatial outputs can be generated by the same autoregressive policy.
    Sec. 2; required for the single-model actor-critic transition.
  • standard math GRPO group-relative advantages plus KL penalty yield stable policy updates for the joint cycle objective.
    Eq. (1) and surrounding text; inherits the standard GRPO formulation.
invented entities (2)
  • CycleGRPO (Actor-as-Own-Critic loop) no independent evidence
    purpose: Joint RL objective that turns localization accuracy into a caption reward without external text labels.
    Core algorithmic contribution; evidence is solely the reported benchmark improvements.
  • quality-aware token-level cycle-consistency reward (R_cap + R_loc) no independent evidence
    purpose: Scalar signal that scores both caption quality and localization trajectories from reconstructive IoU.
    Defined in Sec. 3; no external validation beyond the same experiments that use it.

pith-pipeline@v1.1.0-grok45 · 31122 in / 2454 out tokens · 26544 ms · 2026-07-14T04:36:48.821401+00:00 · methodology

0 comments
read the original abstract

This paper introduces Actor as Its Own Critic, a unified reinforcement learning framework, Cycle Group Relative Policy Optimization (CycleGRPO), that jointly optimizes region understanding and localization for Multimodal Large Language Models (MLLMs). Unlike existing separate pipelines, we leverage the inherent duality between the two tasks to construct a self-evaluating reinforcement learning paradigm: "region $\to$ text $\to$ region''. Specifically, a single MLLM first acts as the actor to generate region captions, then immediately transitions to a critic to ground its generated text back in the spatial domain. Therefore, CycleGRPO requires only region inputs, e.g., masks or bounding boxes, entirely bypassing the need for textual ground truths. A quality-aware token-level cycle-consistency reward is employed to assess the semantic discriminability of text captions via their physical localization accuracy. Empirically, built upon SAMTok, our CycleGRPO framework successfully bootstraps both capabilities simultaneously. Without any task-specific fine-tuning, the framework yields consistent performance gains across a wide range of benchmarks, including region captioning, region VQA, grounded dialogue, and referring segmentation. Overall, CycleGRPO offers a straightforward and scalable way to advance pixel-level capabilities in MLLMs. Code and models are released at https://github.com/devinxzhang/CycleGRPO.

Figures

Figures reproduced from arXiv: 2607.11581 by Haochen Wang, Jason Li, Robby T. Tan, Xin Zhang, Yikang Zhou.

Figure 1
Figure 1. Figure 1: Concept comparison of Reinforcement Learning paradigms for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the CycleGRPO Pipeline. Our framework leverages a single MLLM to instantiate a self-evaluating closed loop. Left (Phase 1: Region Caption￾ing): Acting as the Actor, the model takes an image and a target region M to generate a group of candidate text descriptions {C1, . . . , CG}. Right (Phase 2: Localization Rollout): Symmetrically acting as its own Critic, the exact same model receives each ca… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of region captioning on DLC-Bench [ [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on GRES [28]. We compare CycleGRPO with SAM￾Tok [77] and the ground truth. These examples require advanced spatial reasoning to distinguish targets in complex scenes. In the first row, CycleGRPO accurately localizes multiple disjoint entities mentioned in the query, whereas SAMTok incorrectly includes the man’s coat. In the second row, our model correctly identifies the target despite o… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative results on GroundingSuite [15]. We compare our CycleGRPO with SAMTok [77] and the ground truth. The first row demonstrates part-level seg￾mentation (e.g., the duck’s head), which requires fine-grained understanding of object components. The second row illustrates stuff-class segmentation (e.g., the snow), highlighting the model’s capability for context-aware localization in amorphous regions. C… view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of CycleGRPO compared to the Cap-only and Loc-only GRPO baselines. From left to right: task rewards (CycleGRPO reports captioning and grounding rewards separately as solid and dashed lines), KL divergence from the ref￾erence policy, policy entropy, and gradient norm. CycleGRPO improves both reward components steadily while staying close to the reference policy and maintaining higher entro… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative results on GroundingSuite under the bounding-box out [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablation study on the scaling of training data. reward signal. By maximizing this cycle-consistency reward, CycleGRPO effec￾tively compels the model to bridge its understanding and grounding capabili￾ties, ensuring that every bit of semantic detail is spatially verifiable. This self￾supervised mechanism allows the model to scale its fine-grained intelligence by exploring and rectifying its own internal log… view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of the self-evolution rollouts under the CycleGRPO [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 31 linked inside Pith

  1. [1]

    In: ECCV (2024)

    Ai, Y., Qi, Y., Wang, B., Chen, Y., Wang, X., Tan, R.T.: Domain-adaptive 2d human pose estimation via dual teachers in extremely low-light conditions. In: ECCV (2024)

    2024

  2. [2]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., CycleGRPO 25 Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    arXiv preprint arXiv:2505.14231 (2025)

    Bai, S., Li, M., Liu, Y., Tang, J., Zhang, H., Sun, L., Chu, X., Tang, Y.: Univg-r1: Reasoning guided universal visual grounding with reinforcement learning. arXiv preprint arXiv:2505.14231 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:2505.20272 (2025)

    Cao, M., Zhao, H., Zhang, C., Chang, X., Reid, I., Liang, X.: Ground-r1: In- centivizing grounded visual reasoning via reinforcement learning. arXiv preprint arXiv:2505.20272 (2025)

    arXiv 2025

  6. [6]

    Advances in Neural Information Processing Systems38, 156647–156667 (2026)

    Cao, X., Lin, B., Wang, B., Huang, Z., Tan, R.: 3dot: Texture transfer for 3dgs objects from a single reference image. Advances in Neural Information Processing Systems38, 156647–156667 (2026)

    2026

  7. [7]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, H., Ai, Y., Kim, K., Tan, R.T., Chen, Y., Wang, B.: Udapose: Unsupervised domain adaptation for low-light human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13781– 13792 (2026)

    2026

  8. [8]

    arXiv preprint arXiv:2502.15224 (2025)

    Chen, T., Anumasa, S., Lin, B., Shah, V., Goyal, A., Liu, D.: Auto-bench: An auto- mated benchmark for scientific discovery in llms. arXiv preprint arXiv:2502.15224 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    arXiv preprint arXiv:2510.15614 (2025)

    Chen, T., Lin, B., Yuan, Z., Zou, Q., He, H., Goyal, A., Ong, Y.S., Liu, D.: Hy- pospace: Evaluating llm creativity as set-valued hypothesis generators under un- derdetermination. arXiv preprint arXiv:2510.15614 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    In: ECCV (2024)

    Chen, Y.C., Li, W.H., Sun, C., Wang, Y.C.F., Chen, C.S.: Sam4mllm: Enhance multi-modal large language model for referring expression segmentation. In: ECCV (2024)

    2024

  11. [11]

    DeepMind, G.: Gemini-2.5-pro.https://deepmind.google/models/gemini/pro/ (2025)

    2025

  12. [12]

    arXiv preprint arXiv:2603.17680 (2026)

    Du, W., Yuan, Z., Chen, T., Ke, F., Lin, B., Zhang, S.: Weatherreasonseg: A benchmark for weather-aware reasoning segmentation in visual language models. arXiv preprint arXiv:2603.17680 (2026)

    arXiv 2026

  13. [13]

    In: CVPR

    Guo, Q., De Mello, S., Yin, H., Byeon, W., Cheung, K.C., Yu, Y., Luo, P., Liu, S.: Regiongpt: Towards region understanding vision language model. In: CVPR. pp. 13796–13806 (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2408.16500 (2024)

    Hong, W., Wang, W., Ding, M., Yu, W., Lv, Q., Wang, Y., Cheng, Y., Huang, S., Ji, J., Xue, Z., et al.: Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    arXiv preprint arXiv:2503.10596 (2025)

    Hu, R., Zhu, L., Zhang, Y., Cheng, T., Liu, L., Liu, H., Ran, L., Chen, X., Liu, W.,Wang,X.:Groundingsuite:Measuringcomplexmulti-granularpixelgrounding. arXiv preprint arXiv:2503.10596 (2025)

    Pith/arXiv arXiv 2025

  16. [16]

    arXiv preprint arXiv:2601.09668 (2026)

    Huang, A., Yao, C., Han, C., Wan, F., Guo, H., Lv, H., Zhou, H., Wang, J., Zhou, J., Sun, J., et al.: Step3-vl-10b technical report. arXiv preprint arXiv:2601.09668 (2026)

    arXiv 2026

  17. [17]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    In: CVPR (2024)

    Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: CVPR (2024)

    2024

  19. [19]

    ICLR (2025)

    Lan, M., Chen, C., Zhou, Y., Xu, J., Ke, Y., Wang, X., Feng, L., Zhang, W.: Text4seg: Reimagining image segmentation as text generation. ICLR (2025)

    2025

  20. [20]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024) 26 X

    Lei, Q., Wang, B., Robby T., T.: Ez-hoi: Vlm adaptation via guided prompt learn- ing for zero-shot hoi detection. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024) 26 X. Zhang et al

    2024

  21. [21]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Lei, Q., Wang, B., Tan, R.T.: Hola: Zero-shot hoi detection with low-rank decom- posed vlm feature adaptation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1825–1835 (2025)

    2025

  22. [22]

    arXiv preprint arXiv:2512.17495 (2025)

    Li, R., Li, L., Ren, S., Tian, H., Gu, S., Li, S., Yue, Z., Wang, Y., Ma, W., Yang, Z., et al.: Groundingme: Exposing the visual grounding gap in mllms through multi-dimensional evaluation. arXiv preprint arXiv:2512.17495 (2025)

    arXiv 2025

  23. [23]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Li, S., Tan, L., Tan, R.T.: Bridging day and night: Target-class hallucination sup- pression in unpaired image translation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 6424–6432 (2026)

    2026

  24. [24]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR)

    Li, S., Tan, L., Tan, R.T.: White-balance first, adjust later: Cross-camera color con- stancy via vision-language evaluation. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). pp. 1331–1341 (June 2026)

    2026

  25. [25]

    arXiv preprint arXiv:2506.24102 (2025)

    Li, X., Zhang, T., Li, Y., Yuan, H., Chen, S., Zhou, Y., Meng, J., Sun, Y., Xu, S., Qi, L., et al.: Denseworld-1m: Towards detailed dense grounded caption in the real world. arXiv preprint arXiv:2506.24102 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    arXiv preprint arXiv:2504.16072 (2025)

    Lian, L., Ding, Y., Ge, Y., Liu, S., Mao, H., Li, B., Pavone, M., Liu, M.Y., Dar- rell, T., Yala, A., et al.: Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504.16072 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    arXiv preprint arXiv:2403.20271 (2024)

    Lin, W., Wei, X., An, R., Gao, P., Zou, B., Luo, Y., Huang, S., Zhang, S., Li, H.: Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want. arXiv preprint arXiv:2403.20271 (2024)

    Pith/arXiv arXiv 2024

  28. [28]

    In: CVPR (2023)

    Liu, C., Ding, H., Jiang, X.: Gres: Generalized referring expression segmentation. In: CVPR (2023)

    2023

  29. [29]

    In: NeurIPS (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)

    2023

  30. [30]

    arXiv preprint arXiv:2602.09609 (2026)

    Liu, J., Li, T., Cao, X., Ma, Y., Shang, G., Huang, H., Zhang, C., Chang, X., Huang, Z., Hu, J., et al.: Tele-omni: a unified multimodal framework for video generation and editing. arXiv preprint arXiv:2602.09609 (2026)

    arXiv 2026

  31. [31]

    arXiv preprint arXiv:2503.06520 (2025)

    Liu, Y., Peng, B., Zhong, Z., Yue, Z., Lu, F., Yu, B., Jia, J.: Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520 (2025)

    Pith/arXiv arXiv 2025

  32. [32]

    com / index / introducing - o3 - and - o4 - mini/(2025)

    OpenAI: Openai-o3.https : / / openai . com / index / introducing - o3 - and - o4 - mini/(2025)

    2025

  33. [33]

    Medical & Biological Engineering & Computing62(6), 1821–1836 (2024)

    Qian, Z., Wang, Z., Zhang, X., Wei, B., Lai, M., Shou, J., Fan, Y., Xu, Y.: Msnseg- net: attention-based multi-shape nuclei instance segmentation in histopathology images. Medical & Biological Engineering & Computing62(6), 1821–1836 (2024)

    2024

  34. [34]

    In: CVPR (2024)

    Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R.M., Xing, E., Yang, M.H., Khan, F.S.: Glamm: Pixel grounding large multimodal model. In: CVPR (2024)

    2024

  35. [35]

    arXiv preprint arXiv:2505.23678 (2025)

    Sarch, G., Saha, S., Khandelwal, N., Jain, A., Tarr, M.J., Kumar, A., Fragki- adaki, K.: Grounded reinforcement learning for visual reasoning. arXiv preprint arXiv:2505.23678 (2025)

    Pith/arXiv arXiv 2025

  36. [36]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  37. [37]

    arXiv preprint arXiv:2510.01954 (2025)

    Su, Y., Zhang, H., Li, S., Liu, N., Liao, J., Pan, J., Liu, Y., Xing, X., Sun, C., Li, C., et al.: Patch-as-decodable-token: Towards unified multi-modal vision tasks in mllms. arXiv preprint arXiv:2510.01954 (2025)

    arXiv 2025

  38. [38]

    arXiv preprint arXiv:2606.19534 (2026), https://arxiv.org/abs/2606.19534

    Sun, Y., Wang, Y., Li, J., Tian, Y., Zhang, T., Mai, J., Wang, Y., Wang, H., Bai, J., Yang, L., Tong, Y.: Perceptiondlm: Parallel region perception with CycleGRPO 27 multimodal diffusion language models. arXiv preprint arXiv:2606.19534 (2026), https://arxiv.org/abs/2606.19534

    Pith/arXiv arXiv 2026

  39. [39]

    International Journal of Computer Vision134(2), 72 (2026)

    Tan, L., Li, W., Dai, P., Chen, J., Cao, L., Ji, R.: Knowing where to focus: Attention-guided alignment for text-based person search. International Journal of Computer Vision134(2), 72 (2026)

    2026

  40. [40]

    In: Proceedings of the AAAI conference on artificial intelligence

    Tan, L., Xia, J., Liu, W., Dai, P., Wu, Y., Cao, L.: Occluded person re-identification via saliency-guided patch transfer. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 5070–5078 (2024)

    2024

  41. [41]

    In: Advances in Neural Information Processing Systems

    Tan, L., Zhang, Y., Han, K., Dai, P., Zhang, Y., WU, Y., Ji, R.: Rle: A unified perspective of data augmentation for cross-spectral re-identification. In: Advances in Neural Information Processing Systems. vol. 37, pp. 126977–126996 (2024)

    2024

  42. [42]

    arXiv preprint arXiv:2312.11805 (2023)

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    Pith/arXiv arXiv 2023

  43. [43]

    arXiv preprint arXiv:2507.01006 (2024)

    Team, V., Hong, W., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H., Cheng, J., Qi, J., Ji, J., Pan, L., Duan, S., Wang, W., Wang, Y., Cheng, Y., He, Z., Su, Z., Yang, Z., Pan, Z., Zeng, A., Wang, B., Chen, B., Shi, B., Pang, C., Zhang, C., Yin, D., Yang, F., Chen, G., Xu, J., Zhu, J., Chen, J., Chen, J., Chen, J., Lin, J., Wang, J., Chen, J., Lei, L., Gong, ...

    Pith/arXiv arXiv 2024

  44. [44]

    ICLR (2026)

    Wang, H., Li, X., Huang, Z., Wang, A., Wang, J., Zhang, T., Zheng, J., Bai, S., Kang, Z., Feng, J., et al.: Traceable evidence enhanced visual grounded reasoning: Evaluation and methodology. ICLR (2026)

    2026

  45. [45]

    ICLR (2026)

    Wang, H., Wang, Y., Zhang, T., Zhou, Y., Li, Y., Wang, J., Tian, Y., Meng, J., Huang, Z., Mai, G., Wang, A., Tong, Y., Wang, Z., Li, X., Zhang, Z.: Grasp any region: Towards precise, contextual pixel understanding for multimodal llms. ICLR (2026)

    2026

  46. [46]

    arXiv preprint arXiv:2510.18876 (2025)

    Wang, H., Wang, Y., Zhang, T., Zhou, Y., Li, Y., Wang, J., Tian, Y., Meng, J., Huang, Z., Mai, G., et al.: Grasp any region: Towards precise, contextual pixel understanding for multimodal llms. arXiv preprint arXiv:2510.18876 (2025)

    arXiv 2025

  47. [47]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, H., Zhao, Y., Wang, T., Fan, H., Zhang, X., Zhang, Z.: Ross3d: Reconstruc- tive visual instruction tuning with 3d-awareness. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9275–9286 (2025)

    2025

  48. [48]

    In: ICLR (2025)

    Wang, H., Zheng, A., Zhao, Y., Wang, T., Ge, Z., Zhang, X., Zhang, Z.: Recon- structive visual instruction tuning. In: ICLR (2025)

    2025

  49. [49]

    arXiv preprint arXiv:2506.11991 (2025)

    Wang, J., Kang, Z., Wang, H., Jiang, H., Li, J., Wu, B., Wang, Y., Ran, J., Liang, X., Feng, C., et al.: Vgr: Visual grounded reasoning. arXiv preprint arXiv:2506.11991 (2025)

    Pith/arXiv arXiv 2025

  50. [50]

    arXiv preprint arXiv:2508.04107 (2025)

    Wang, J., Wu, Z., Huang, D., Zheng, Y., Wang, H.: Unlocking the potential of mllms in referring expression segmentation via a light-weight mask decoder. arXiv preprint arXiv:2508.04107 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    arXiv preprint arXiv:2505.16495 (2025) 28 X

    Wang, L., Lin, H., Chen, S., Wang, T., Cheng, C., Zhong, Y., Zheng, D., Zhao, W.: Alto: Adaptive-length tokenizer for autoregressive mask generation. arXiv preprint arXiv:2505.16495 (2025) 28 X. Zhang et al

    arXiv 2025

  52. [52]

    In: ICCV (2025)

    Wang, T., Cheng, C., Wang, L., Chen, S., Zhao, W.: Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model. In: ICCV (2025)

    2025

  53. [53]

    arXiv preprint arXiv:2508.18265 (2025)

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025)

    Pith/arXiv arXiv 2025

  54. [54]

    NeurIPS (2023)

    Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al.: Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. NeurIPS (2023)

    2023

  55. [55]

    In: NeurIPS (2025)

    Wang, X., Ru, L., Huang, Z., Ji, K., Zheng, D., Chen, J., Zhou, J.: Argenseg: Image segmentation with autoregressive image generation model. In: NeurIPS (2025)

    2025

  56. [56]

    In: ICCV

    Wei, C., Zhong, Y., Tan, H., Zeng, Y., Liu, Y., Wang, H., Yang, Y.: Instructseg: Unifying instructed visual segmentation with multi-modal large language models. In: ICCV. pp. 20193–20203 (2025)

    2025

  57. [57]

    arXiv preprint arXiv:2412.10302 (2024)

    Wu,Z.,Chen,X.,Pan,Z.,Liu,X.,Liu,W.,Dai,D.,Gao,H.,Ma,Y.,Wu,C.,Wang, B., et al.: Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024)

    Pith/arXiv arXiv 2024

  58. [58]

    arXiv preprint arXiv:2509.22647 (2025)

    Xing, L., Dong, X., Zang, Y., Cao, Y., Liang, J., Huang, Q., Wang, J., Wu, F., Lin, D.: Caprl: Stimulating dense image caption capabilities via reinforcement learning. arXiv preprint arXiv:2509.22647 (2025)

    arXiv 2025

  59. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Yan, W., Li, H., Xu, H., Ye, N., Ai, Y., Liu, S., Hu, J.: Las-comp: Zero-shot 3d completion with latent-spatial consistency. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7588–7599 (June 2026)

    2026

  60. [60]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)

    Yan,W.,Li,M.,Li,H.,Shao,S.,Tan,R.T.:Synthetic-to-realself-supervisedrobust depth estimation via learning with motion and structure priors. In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). pp. 21880– 21890 (June 2025)

    2025

  61. [61]

    arXiv preprint arXiv:2509.00665 (2025)

    Yan, W., Zhang, X., Tan, R.T.: Er-lora: Effective-rank guided adaptation for weather-generalized depth estimation. arXiv preprint arXiv:2509.00665 (2025)

    Pith/arXiv arXiv 2025

  62. [62]

    In: Proceedings of the AAAI Con- ference on Artificial Intelligence

    Yang, X., Zhang, X., Wang, X.: Erf: A benchmark dataset for robust semantic segmentation under extreme rainfall conditions. In: Proceedings of the AAAI Con- ference on Artificial Intelligence. vol. 39, pp. 9301–9309 (2025)

    2025

  63. [63]

    arXiv preprint arXiv:2506.22624 (2025)

    You, Z., Wu, Z.: Seg-r1: Segmentation can be surprisingly simple with reinforce- ment learning. arXiv preprint arXiv:2506.22624 (2025)

    Pith/arXiv arXiv 2025

  64. [64]

    arXiv preprint arXiv:2501.04001 (2025)

    Yuan, H., Li, X., Zhang, T., Huang, Z., Xu, S., Ji, S., Tong, Y., Qi, L., Feng, J., Yang, M.H.: Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001 (2025)

    Pith/arXiv arXiv 2025

  65. [65]

    In: CVPR (2024)

    Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., Zhu, J.: Osprey: Pixel understanding with visual instruction tuning. In: CVPR (2024)

    2024

  66. [66]

    In: CVPR (2025)

    Yuan, Y., Zhang, H., Li, W., Cheng, Z., Zhang, B., Li, L., Li, X., Zhao, D., Zhang, W., Zhuang, Y., et al.: Videorefer suite: Advancing spatial-temporal object under- standing with video llm. In: CVPR (2025)

    2025

  67. [67]

    arXiv preprint arXiv:2510.23603 (2025)

    Yuan, Y., Zhang, W., Li, X., Wang, S., Li, K., Li, W., Xiao, J., Zhang, L., Ooi, B.C.: Pixelrefer: A unified framework for spatio-temporal object referring with arbitrary granularity. arXiv preprint arXiv:2510.23603 (2025)

    arXiv 2025

  68. [68]

    arXiv preprint arXiv:2404.07973 (2024)

    Zhang, H., You, H., Dufter, P., Zhang, B., Chen, C., Chen, H.Y., Fu, T.J., Wang, W.Y., Chang, S.F., Gan, Z., et al.: Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 (2024)

    Pith/arXiv arXiv 2024

  69. [69]

    NeurIPS (2024) CycleGRPO 29

    Zhang, T., Li, X., Fei, H., Yuan, H., Wu, S., Ji, S., Loy, C.C., Yan, S.: Omg- llava: Bridging image-level, object-level, pixel-level reasoning and understanding. NeurIPS (2024) CycleGRPO 29

    2024

  70. [70]

    arXiv preprint arXiv:2504.10465 (2025)

    Zhang, T., Li, X., Huang, Z., Li, Y., Lei, W., Deng, X., Chen, S., Ji, S., Feng, J.: Pixel-sail: Single transformer for pixel-grounded understanding. arXiv preprint arXiv:2504.10465 (2025)

    Pith/arXiv arXiv 2025

  71. [71]

    arXiv preprint arXiv:2507.04664 (2025)

    Zhang, T., Wei, S., Chen, S., Yu, W., Luo, M., Ji, S.: Vectorllm: Human-like extraction of structured building contours vis multimodal llms. arXiv preprint arXiv:2507.04664 (2025)

    Pith/arXiv arXiv 2025

  72. [72]

    IEEE Transactions on Image Processing32, 4247–4258 (2023)

    Zhang, X., Chen, Y.C.: Adaptive domain generalization via online disagreement minimization. IEEE Transactions on Image Processing32, 4247–4258 (2023)

    2023

  73. [73]

    Zhang, X., Tan, R.T.: Mamba as a bridge: Where vision foundation models meet visionlanguagemodelsfordomain-generalizedsemanticsegmentation.In:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14527–14537 (2025)

    2025

  74. [74]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhang, X., Xie, J., Yuan, Y., Mi, M.B., Tan, R.T.: Heap: unsupervised object discovery and localization with contrastive grouping. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 7323–7331 (2024)

    2024

  75. [75]

    arXiv preprint arXiv:2602.03890 (2026)

    Zhang, X., Yan, W., Shi, Y., Qiu, X., He, T., Li, Y., Li, M., Fan, H.: 4dpc2hat: To- wards dynamic point cloud understanding with failure-aware bootstrapping. arXiv preprint arXiv:2602.03890 (2026)

    Pith/arXiv arXiv 2026

  76. [76]

    arXiv preprint arXiv:2406.20076 (2024)

    Zhang, Y., Cheng, T., Zhu, L., Hu, R., Liu, L., Liu, H., Ran, L., Chen, X., Liu, W., Wang, X.: Evf-sam: Early vision-language fusion for text-prompted segment anything model. arXiv preprint arXiv:2406.20076 (2024)

    Pith/arXiv arXiv 2024

  77. [77]

    arXiv preprint arXiv:2601.16093 (2026)

    Zhou, Y., Zhang, T., Gong, D., Wu, Y., Tian, Y., Wang, H., Yuan, H., Wang, J., Qi, L., Fei, H., et al.: Samtok: Representing any mask with two words. arXiv preprint arXiv:2601.16093 (2026)

    arXiv 2026

  78. [78]

    arXiv preprint arXiv:2505.15810 (2025)

    Zhou, Y., Dai, S., Wang, S., Zhou, K., Jia, Q., Xu, J.: Gui-g1: Understand- ing r1-zero-like training for visual grounding in gui agents. arXiv preprint arXiv:2505.15810 (2025)

    Pith/arXiv arXiv 2025

  79. [79]

    arXiv preprint arXiv:2504.10479 (2025)

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025)

    Pith/arXiv arXiv 2025

  80. [80]

    arXiv preprint arXiv:2508.14153 (2025)

    Zhu, L., Ouyang, B., Zhang, Y., Cheng, T., Hu, R., Shen, H., Ran, L., Chen, X., Yu, L., Liu, W., et al.: Lens: Learning to segment anything with unified reinforced reasoning. arXiv preprint arXiv:2508.14153 (2025)

    arXiv 2025