Pith. sign in

REVIEW 3 major objections 4 minor 141 references

Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A calibrated reinforcement learning method lets multimodal LLMs refuse nonexistent objects while keeping their localization accuracy.

desk verdict A genuinely novel RL recipe for teaching MLLMs to refuse, but the paper's central claim of preserving localization is undercut by the gRefCOCO P-acc it never reports. read the letter →

arxiv 2608.04698 v1 pith:QVBZE64L submitted 2026-08-05 cs.CV cs.AI

Xuzheng Yang , Jun Ling , Tao Huang , Caiyan Qin , Peng Wang This is my paper
classification cs.CVcs.AI
keywords GeneralizedReferringExpressionComprehensionrefusalcalibrationGRPOmultimodallargelanguagemodelsvisualgroundinghallucinationreinforcementlearningnegativesamples
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a multimodal large language model trained for referring expression comprehension can be post-trained to say 'no' — to output a refusal instead of a bounding box when the described object does not exist — without losing its ability to localize objects that do exist. The problem is that standard supervised fine-tuning and reinforcement learning improve refusal but sharply degrade positive-sample localization, because negatives are easier to optimize. The proposed Refusal-Calibrated GRPO answer this by forcing a 'None' rollout on negative samples during training, giving the policy a valid reward signal, and then calibrating the optimization with a contrastive reward and negative-sample advantage scaling so the model does not over-refuse. On three GREC benchmarks, the method reports higher overall precision than existing specialist and MLLM baselines while retaining a strong refusal rate, suggesting the trade-off is not inherent.

What carries the argument

The machinery is a two-stage reinforcement learning post-training recipe built on GRPO. Stage I has three components: forced-rejection rollout, which injects a 'None' completion when the model does not produce one so negative samples yield a nonzero advantage; a contrastive reward, which gives $+1$ for $IoU \ge 0.5$, $0$ for a wrong positive box, and $-1$ for predicting 'None' on a positive sample, while granting $+1$ for a correct 'None' on a negative sample; and negative advantage scaling, which multiplies the negative-sample advantage by a factor $\alpha \in (0,1]$ to balance the easier refusal task. Stage II reinforces explanation generation with a rationale reward that checks whether the generated refusal explanation mentions the incorrect and the original semantic patterns, rewarding causal accounts of absence. This combination is what carries the argument: it makes negative samples optimizable for GRPO while counteracting the optimization imbalance that makes plain SFT and GRPO collapse into over-refusal.

What would settle it

Train RC-GRPO on FineCops-Ref, then evaluate with forced rejection turned off (free sampling only) and compare N-acc and P-acc to the reported numbers; if N-acc collapses toward the base model while P-acc stays high, the refusal behavior is an artifact of the forced rollout rather than learned policy. Alternatively, compute the KL divergence between the forced-rollout distribution and the current policy at the end of training; a large remaining divergence would indicate the off-policy bias has not decayed as claimed.

Watch

Extended reading notes

Core claim

The central claim is that GRPO-style reinforcement learning can teach a multimodal LLM to reject nonexistent referring expressions if the rollout distribution is calibrated: when no completion in the group contains 'None', the paper forces a constrained beam-search output of 'There are none' and computes the advantage using the sequence probability under the current logits. This forced-rejection rollout prevents the advantage estimate from vanishing for negative samples, a failure mode of standard GRPO. To keep the model from drifting toward the easy 'None' answer, positives get a contrastive reward that penalizes false refusals, and negative-sample advantages are scaled down. A second stage adds a rule-based rationale reward for explaining why the object is absent, consolidating causal refusal. The paper claims this yields a superior Pareto frontier of positive accuracy versus negative refusal accuracy on FineCops-Ref, gRefCOCO, and D3, with negligible degradation on general multimodal benchmarks.

Load-bearing premise

The load-bearing premise is that injecting a forced 'None' rollout during training yields GRPO advantage estimates that are valid enough — and whose off-policy bias decays fast enough — that the policy actually learns refusal rather than merely imitating the injection; the paper provides a decay plot but no quantitative bound on the residual bias.

Editorial extensions

If this is right

  • A post-trained 4B or 7B multimodal LLM can reach or exceed the refusal reliability of specialized detector-plus-MLLM pipelines while keeping precision within a few points of the base grounding model.
  • The $\alpha=0.5$ scaling factor is reported as a universal safe default across datasets and backbones, so practitioners can apply RC-GRPO without per-task tuning.
  • The forced-rejection rollout removes the need for an external teacher model or SFT cold start, making the approach self-contained and applicable to any base MLLM that can emit a refusal token.
  • The second-stage rationale reward improves the faithfulness and logical validity of refusal explanations, which matters if the model is to act in robotics or embodied settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit is that the same calibration recipe could transfer to other abstention problems — open-set detection, out-of-distribution classification, or any generative task where a model should decline to answer — since the mechanism only needs a token-level refusal and a reward that penalizes false abstention.
  • The forced-rejection rollout's off-policy bias is acknowledged but not bounded; a useful testable extension is to measure the intervention rate and the KL divergence between the rollout distribution and the policy late in training, and to verify that refusal persists when forced rollouts are disabled at evaluation.
  • The rationale reward is defined by the dataset's construction logic, so applying the second stage to a new dataset requires either the same construction metadata or a learned verifier; the paper's zero-shot gains on non-template benchmarks suggest the first stage carries most of the transferable effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes RC-GRPO, a two-stage reinforcement-learning post-training method for Generalized Referring Expression Comprehension (GREC). Stage I combines forced-rejection rollouts (constrained beam search that forces a "There are none." output on negative samples), a contrastive reward that penalizes false refusals on positive samples, and a negative-advantage scaling factor \alpha. Stage II adds a rule-based rationale reward that rewards explanations identifying the specific attribute/relation mismatch. The method is evaluated on FineCops-Ref, gRefCOCO, and D3 with Qwen2.5-VL-7B and Qwen3-VL-4B backbones, reporting higher Precision and N-acc than SFT, DPO, GRPO, and the CRS baseline, together with ablations, sensitivity analysis, data-scaling experiments, and an appendix discussing the off-policy bias of forced rejection.

Significance. If the claims hold, this is a practically useful recipe for teaching MLLMs to abstain from hallucinated bounding boxes without severe localization loss, with broader relevance to hallucination mitigation and trustworthy visual grounding. The paper has several strengths: it studies an underexplored and practically important task; it evaluates on three benchmarks, including two external ones (gRefCOCO, D3) that mitigate the in-group nature of FineCops-Ref; it provides component ablations, an \alpha-sensitivity study, a data-scaling trend, and an explicit appendix that acknowledges the off-policy bias rather than hiding it. The 2K-sample training set shows encouraging sample efficiency. However, the main evidence for the headline claim of "preserving localization performance" is incomplete, because P-acc is not reported for gRefCOCO and the reported Precision/N-acc values imply a substantial P-acc drop under a balanced split; the ablation of the forced-rejection component contradicts the paper's own motivation; and the off-policy bias is acknowledged but not quantified. No code, seeds, or error bars are provided, making the single-run results hard to assess.

major comments (3)
  1. [Method, Forced-Rejection Rollout; Appendix, On the Training Bias of Forced Rejection] The claim that "a higher Pr indicates a strictly superior model" is mathematically unjustified once Pr is a weighted sum of P-acc and N-acc, because a model with lower P-acc can have higher Pr solely by raising N-acc. More importantly, P-acc is never reported for gRefCOCO. If the gRefCOCO split is balanced (50/50), Table 1 implies for Qwen2.5-VL-7B: base P-acc is about 70.0/91.0/72.4 on val/test-A/test-B, while RC-GRPO-I gives about 58.0/58.8/48.1 — a 12–32 point localization drop. This directly contradicts the abstract's "preserving localization performance" and the text's claim that RC-GRPO attains "superior localization accuracy." The authors must report P-acc for all gRefCOCO splits, state the exact positive/negative ratio used to define Pr, and, if the implied drop is real, revise the headline claim to describe a trade-off rather than preservation.
  2. [Table 3] The paper enforces "None" outputs during negative-sample rollouts via constrained beam search, but then computes sequence probabilities under the current policy and estimates advantages "exactly like standard trajectories." This is an off-policy estimator: the rollouts are not sampled from \pi_\theta, so the resulting GRPO advantages are biased. The appendix explicitly acknowledges this bias but provides no quantitative bound, no importance-weighting correction, and only a qualitative decay curve (Figure 4). Because the method's central novelty is exactly this forced-rejection mechanism for "valid advantage estimation," the authors should either provide a corrected estimator, a formal bound on the bias, or an empirical analysis (e.g., comparing against an on-policy variant with longer exploration or an importance-weighted variant) showing that the bias does not drive the reported refusal gains.
  3. [Table 3] Ablation "w/o Forced-Rejection Rollout" achieves Pr 65.5, P-acc 68.0, N-acc 58.4, which is nearly identical to RC-GRPO-I (Pr 66.6, P-acc 69.0, N-acc 59.5). This directly contradicts the text's assertion that without forced rejection the model "cannot compute valid advantages for negatives" and that learning quickly stalls. If the ablation is correctly labeled, the forced-rejection component is not essential to the reported balance, undermining the paper's motivation. The authors need to reconcile this discrepancy, either by explaining how the model obtains meaningful negative-sample gradients without forced rejection or by correcting the ablation and the surrounding discussion.
minor comments (4)
  1. [Experiments, Evaluation Settings] Several numerical comparisons in the text do not match Table 1. For example, the text cites gRefCOCO Precision improvements of "53.5→60.5" and "48.8→60.5" for Qwen2.5-7B, but Table 1 shows RC-GRPO-I at 62.8 on the gRefCOCO val split; the later "53.5→70.6" and "50.1→70.6" also do not correspond to any visible table entry. Please align all numbers between the text and Table 1.
  2. [Appendix, Additional Analyses] No experimental seeds, error bars, or multi-run variance are reported for any table. Given the small 2K training set and the sensitivity of RL training, the authors should report at least two or three seeds for the main comparisons, or explicitly state that results are single-run observations and interpret them as such.
  3. [Training Data] The appendix references Figures 4, 5, and 6 (forced-rejection trigger rate, training dynamics, and \alpha trend), but these figures are not present in the submitted main text. Please ensure they are included in the final version and referenced by number.
  4. [Training Data] The main text says "2,000 high-quality samples" without defining "high-quality"; the appendix later clarifies it means excluding corrupted annotations via stratified random sampling. Please state this definition in the main text to avoid the impression of cherry-picking.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central RC-GRPO gains are externally grounded on gRefCOCO and D3; the FineCops-Ref overlap is a training-data choice, not a derived prediction.

full rationale

The claimed derivation chain is not circular. RC-GRPO is a post-training RL method whose optimization signals come from rollouts and hand-specified rewards (Eqs. 1–5), not from the evaluation benchmarks; its advantages are not defined in terms of gRefCOCO or D3 metrics, and the headline improvements are reproduced on those external benchmarks rather than only on the same-group FineCops-Ref set. The FineCops-Ref citation (Liu et al. 2024a, which includes present authors Yang and Wang) is a same-group training/evaluation resource, but it is used as data and construction-annotation supervision, not as a theorem or as a fitted target, so it does not force the result. The CRS baseline (Yang et al. 2025c) is also same-group, but it is re-trained on the paper's dataset and outperformed, so no load-bearing self-citation chain is present. The appendix's admission of residual off-policy bias from forced rejection is a real limitation but not a circular step, since the intervention rate decays and the policy is updated with current-logit sequence probabilities. One genuine reporting gap remains: the paper nowhere tabulates gRefCOCO P-acc even though it defines Pr as a weighted sum of P-acc and N-acc, so the claim of “preserving localization performance” on gRefCOCO is not fully certified there; this is missing support, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The main load-bearing overhead is a set of hand-set reward constants and an off-policy rollout validity assumption. No new physical or architectural entities are introduced; the paper's contribution is an RL training recipe.

free parameters (4)
  • Negative advantage scaling factor alpha = 0.5
    Selected by a sensitivity sweep on FineCops-Ref (Table 4) as the best trade-off; then used for all benchmarks.
  • Contrastive reward magnitudes for positive samples = 1 (IoU>=0.5), 0 (IoU<0.5), -1 (predicts None)
    Hand-chosen reward shaping in Eq. (1); no search or justification beyond balancing optimization.
  • Negative sample reward magnitudes = 1 (None), 0 (otherwise)
    Hand-chosen in Eq. (2); no independent calibration.
  • Rationale reward weights w_inc, w_orig, r_reject = 0.5, 0.3, 0.2
    Set following FineCops-Ref construction logic in Eqs. (4)-(5); manually chosen, not derived from data.
assumptions (4)
  • domain assumption Forced 'None' trajectories can be treated as valid rollouts if sequence probability is computed under current logits, and the off-policy bias decays during training.
    Claimed in the appendix 'On the Training Bias of Forced Rejection'; no formal bound is given, but the GRPO advantage estimates depend on this.
  • domain assumption FineCops-Ref training and test splits are disjoint, and training on the 2K subset does not leak into test evaluation.
    The paper does not describe the exact split identity; the primary benchmark is the authors' own dataset.
  • domain assumption The Pinc and Porig semantic patterns recorded during FineCops-Ref construction are correct gold labels for the rationale reward and are available for all training negatives.
    Used in Eq. (4) for the Stage II reasoning reward; only well-defined for fine-grained synthetic negatives.
  • domain assumption IoU>=0.5 on serialized bounding-box coordinates is a faithful proxy for correct localization.
    Standard REC evaluation convention, but the paper does not analyze coordinate tokenization effects on IoU computation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO." pith.science (2026). https://pith.science/paper/QVBZE64L

@misc{pith2026260804698,
  author       = {Pith},
  title        = {Pith review of: Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVBZE64L}},
  note         = {Machine review of arXiv:2608.04698}
}
read the original abstract

We tackle the challenging yet underexplored task of Generalized Referring Expression Comprehension (GREC), which requires a model to localize the object described by a textual expression when it exists (positive sample) and to refuse output when it does not (negative sample). Although Multimodal Large Language Models (MLLMs) excel at localizing existing objects, they often fail to reject nonexistent ones due to the absence of negative samples during training, producing hallucinated bounding boxes. Existing post-training approaches such as supervised fine-tuning (SFT) and reinforcement learning (RL) enhance refusal behavior but usually degrade localization accuracy on positive samples, undermining the model's core competence. To address this, we propose Refusal-Calibrated Group Relative Policy Optimization (RC-GRPO), a calibrated RL strategy that strengthens the refusal ability of MLLMs while preserving localization performance. It enforces "None" outputs in rollouts for valid advantage estimation on negative samples and applies a penalty to prevent over-refusal on positives, achieving a balanced trade-off between accuracy and reliability. A second-stage reasoning reinforcement further consolidates causal understanding and interpretability. Experiments on three GREC benchmarks demonstrate that RC-GRPO attains superior localization accuracy while maintaining strong refusal capability.

Figures

Figures reproduced from arXiv: 2608.04698 by the authors.

Figure 1
Figure 1. Existing MLLMs fail to reject nonexistent cases. In the upper example, when the referred object is absent, Qwen2.5-VL-32B (Bai et al. 2025a) incorrectly localizes an unrelated region. In the lower example, Qwen3-VL￾30B (Yang et al. 2025a) overlooks an attribute mismatch and still outputs a bounding box. In contrast, our method successfully identifies the absence of the referred target and predicts a refusal with cle… view at source ↗
Figure 2
Figure 2. The pipeline of RC-GRPO. The left side shows examples of model input and output. Stage I (Refusal Calibration with Contrastive Reward) equips the model with balanced rejection ability by enforcing “None” outputs and applying contrastive rewards to preserve localization on positives. Stage II (Reason-Guided Reinforcement) further strengthens this behavior by encouraging the model to reason about the cause of absence,… view at source ↗
Figure 3
Figure 3. Pareto frontier of P-acc vs. N-acc on gRefCOCO. Baselines like CRS sacrifice grounding accuracy (P-acc) to gain refusal reliability (N-acc), leading to a rapid entry into the utility collapse zone. SFT exhibits poor generalization; despite its high N-acc on FineCops-Ref, its negligible refusal ability here (12.8%) underscores a failure to learn robust logic. In contrast, RC-GRPO establishes a superior Pareto frontie… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The forced-rejection trigger rate decays naturally [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Accuracy changes of SFT, GRPO, and RC-GRPO [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Trend of P-acc and N-acc as the scaling factor [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results on the GREC task. Compared methods include Qwen2.5-VL (Bai et al. 2025a), SFT, GRPO, CRS (Yang et al. 2025c), and our RC-GRPO. In each referring expression, green text marks the main target and red text highlights invalid referring cues. Predicted o…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

141 extracted references · 29 canonical work pages

  1. [1]

    ICLR , year=

    Ferret: Refer and Ground Anything Anywhere at Any Granularity , author=. ICLR , year=

  2. [2]

    CVPR , pages=

    Gqa: A new dataset for real-world visual reasoning and compositional question answering , author=. CVPR , pages=

  3. [3]

    NAACL , year=

    Visual referring expression recognition: What do systems actually learn? , author=. NAACL , year=

  4. [4]

    ICCV (ICCV) , month =

    Wu, Yu and Wei, Yana and Wang, Haozhe and Liu, Yongfei and Yang, Sibei and He, Xuming , title =. ICCV (ICCV) , month =. 2023 , pages =

    2023

  5. [5]

    arXiv preprint arXiv:2304.03246 , year=

    Inst-inpaint: Instructing to remove objects with diffusion models , author=. arXiv preprint arXiv:2304.03246 , year=

    arXiv

  6. [6]

    IEEE TPAMI , year=

    Improving generalized visual grounding with instance-aware joint learning , author=. IEEE TPAMI , year=

  7. [7]

    ACM MM , year=

    3d-gres: Generalized 3d referring expression segmentation , author=. ACM MM , year=

  8. [8]

    arXiv preprint arXiv:2502.13923 , year=

    Qwen2.5-VL Technical Report , author=. arXiv preprint arXiv:2502.13923 , year=

    arXiv

Show all 141 references
  1. [9]

    arXiv preprint arXiv:2409.12191 , year=

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution , author=. arXiv preprint arXiv:2409.12191 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:2308.12966 , year=

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond , author=. arXiv preprint arXiv:2308.12966 , year=

    arXiv

  3. [11]

    IEEE TPAMI , year=

    New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration , author=. IEEE TPAMI , year=

  4. [12]

    arXiv preprint arXiv:2401.04088 , year=

    Mixtral of experts , author=. arXiv preprint arXiv:2401.04088 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2306.13549 , year=

    A survey on multimodal large language models , author=. arXiv preprint arXiv:2306.13549 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2005.14165 , year=

    Language models are few-shot learners , author=. arXiv preprint arXiv:2005.14165 , year=

    2005 arXiv

  7. [15]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    arXiv

  8. [16]

    arXiv preprint arXiv:2506.08008 , year=

    Hidden in plain sight: Vlms overlook their visual representations , author=. arXiv preprint arXiv:2506.08008 , year=

    arXiv

  9. [17]

    Proceedings of naacL-HLT , volume=

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of naacL-HLT , volume=. 2019 , organization=

    2019

  10. [18]

    CVPR , pages=

    Florence-2: Advancing a unified representation for a variety of vision tasks , author=. CVPR , pages=

  11. [19]

    CVPR , year=

    ROD-MLLM: Towards More Reliable Object Detection in Multimodal Large Language Models , author=. CVPR , year=

  12. [20]

    arXiv preprint arXiv:2404.07973 , year=

    Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models , author=. arXiv preprint arXiv:2404.07973 , year=

    arXiv

  13. [21]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  14. [22]

    arXiv preprint arXiv:2308.16182 , year=

    Grec: Generalized referring expression comprehension , author=. arXiv preprint arXiv:2308.16182 , year=

    arXiv

  15. [23]

    CVPR , year=

    Gres: Generalized referring expression segmentation , author=. CVPR , year=

  16. [24]

    CVPR , year=

    Revisiting counterfactual problems in referring expression comprehension , author=. CVPR , year=

  17. [25]

    ECCV , year=

    Modeling context between objects for referring expression understanding , author=. ECCV , year=

  18. [26]

    The Twelfth International Conference on Learning Representations , year=

    Grounding Multimodal Large Language Models to the World , author=. The Twelfth International Conference on Learning Representations , year=

  19. [27]

    arXiv preprint arXiv:2312.03594 , year=

    A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting , author=. arXiv preprint arXiv:2312.03594 , year=

    arXiv

  20. [28]

    ICCV , year=

    Dire for diffusion-generated image detection , author=. ICCV , year=

  21. [29]

    arXiv preprint arXiv:2304.07193 , year=

    Dinov2: Learning robust visual features without supervision , author=. arXiv preprint arXiv:2304.07193 , year=

    arXiv

  22. [30]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  23. [31]

    arXiv preprint arXiv:2401.06071 , year=

    LEGO: Language Enhanced Multi-modal Grounding Model , author=. arXiv preprint arXiv:2401.06071 , year=

    arXiv

  24. [32]

    arXiv preprint arXiv:2306.01879 , year=

    Visualgptscore: Visio-linguistic reasoning with multimodal generative pre-training scores , author=. arXiv preprint arXiv:2306.01879 , year=

    arXiv

  25. [33]

    Junyan Li and Delin Chen and Yining Hong and Zhenfang Chen and Peihao Chen and Yikang Shen and Chuang Gan , booktitle=. Co. 2024 , url=

    2024

  26. [34]

    arXiv preprint arXiv:2312.12423 , year=

    Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model , author=. arXiv preprint arXiv:2312.12423 , year=

    arXiv

  27. [35]

    CVPR , year =

    Tong, Shengbang and Liu, Zhuang and Zhai, Yuexiang and Ma, Yi and LeCun, Yann and Xie, Saining , title =. CVPR , year =

  28. [36]

    CVPR , pages=

    Clevr-ref+: Diagnosing visual reasoning with referring expressions , author=. CVPR , pages=

  29. [37]

    arXiv preprint arXiv:2403.16999 , year=

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models , author=. arXiv preprint arXiv:2403.16999 , year=

    arXiv

  30. [38]

    arXiv preprint arXiv:2404.13013 , year=

    Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models , author=. arXiv preprint arXiv:2404.13013 , year=

    arXiv

  31. [39]

    ICCV , year=

    Refego: Referring expression comprehension dataset from first-person perception of ego4d , author=. ICCV , year=

  32. [40]

    International Conference on Machine Learning , pages=

    Detectgpt: Zero-shot machine-generated text detection using probability curvature , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  33. [41]

    CVPR , pages=

    Crepe: Can vision-language foundation models reason compositionally? , author=. CVPR , pages=

  34. [42]

    ICCV , year=

    Omnilabel: A challenging benchmark for language-based object detection , author=. ICCV , year=

  35. [43]

    and Chandraker, Manmohan and Schulter, Samuel , title =

    Zhao, Shiyu and Zhao, Long and G, Vijay Kumar B and Suh, Yumin and Metaxas, Dimitris N. and Chandraker, Manmohan and Schulter, Samuel , title =. CVPR (CVPR) , month =. 2024 , pages =

    2024

  36. [44]

    International Conference on Learning Representations , year=

    Open-Set Recognition: A Good Closed-Set Classifier is All You Need , author=. International Conference on Learning Representations , year=

  37. [45]

    CVPR , pages=

    Winoground: Probing vision and language models for visio-linguistic compositionality , author=. CVPR , pages=

  38. [46]

    NeurIPS , volume=

    A benchmark for compositional visual reasoning , author=. NeurIPS , volume=

  39. [47]

    NeurIPS , year=

    Mme: A comprehensive evaluation benchmark for multimodal large language models , author=. NeurIPS , year=

  40. [48]

    Advances in

    Wang, Wenhai and Chen, Zhe and Chen, Xiaokang and Wu, Jiannan and Zhu, Xizhou and Zeng, Gang and Luo, Ping and Lu, Tong and Zhou, Jie and Qiao, Yu and Dai, Jifeng , editor =. Advances in. 2023 , pages =

    2023

  41. [49]

    arXiv preprint arXiv:2408.13257 , year=

    MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? , author=. arXiv preprint arXiv:2408.13257 , year=

    arXiv

  42. [50]

    CVPR , year=

    Modeling relationships in referential expressions with compositional modular networks , author=. CVPR , year=

  43. [51]

    CVPR , year=

    Grounding referring expressions in images by variational context , author=. CVPR , year=

  44. [52]

    CVPR , year=

    Parallel attention: A unified framework for visual object discovery through dialogs and queries , author=. CVPR , year=

  45. [53]

    CVPR , year=

    Mattnet: Modular attention network for referring expression comprehension , author=. CVPR , year=

  46. [54]

    CVPR , year=

    Improving referring expression grounding with cross-modal attention-guided erasing , author=. CVPR , year=

  47. [55]

    ICCV , year=

    Dynamic graph attention for referring expression comprehension , author=. ICCV , year=

  48. [56]

    Learning to compose and reason with language tree structures for visual grounding , author=

  49. [57]

    ICCV , year=

    Learning to assemble neural module tree networks for visual grounding , author=. ICCV , year=

  50. [58]

    European Conference on Computer Vision , year=

    X-detr: A versatile architecture for instance-wise vision-language tasks , author=. European Conference on Computer Vision , year=

  51. [59]

    NeurIPS , year=

    Referring transformer: A one-step approach to multi-task visual grounding , author=. NeurIPS , year=

  52. [60]

    CVPR , year=

    A real-time cross-modality correlation filtering method for referring expression comprehension , author=. CVPR , year=

  53. [61]

    ICCV , year=

    A fast and accurate one-stage approach to visual grounding , author=. ICCV , year=

  54. [62]

    ICCV , year=

    Trar: Routing the attention spans in transformer for visual question answering , author=. ICCV , year=

  55. [63]

    ICCV , year=

    Mdetr-modulated detection for end-to-end multi-modal understanding , author=. ICCV , year=

  56. [64]

    arXiv preprint arXiv:2409.17531 , year=

    SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion , author=. arXiv preprint arXiv:2409.17531 , year=

    arXiv

  57. [65]

    ICCV , year=

    Transvg: End-to-end visual grounding with transformers , author=. ICCV , year=

  58. [66]

    IEEE Transactions on Neural Networks and Learning Systems , year=

    A real-time global inference network for one-stage referring expression comprehension , author=. IEEE Transactions on Neural Networks and Learning Systems , year=

  59. [67]

    CVPR , year=

    Multi-task collaborative network for joint referring expression comprehension and segmentation , author=. CVPR , year=

  60. [68]

    Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIV 16 , year=

    Improving one-stage visual grounding by recursive sub-query construction , author=. Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIV 16 , year=

    2020

  61. [69]

    CVPR , year=

    Look before you leap: Learning landmark features for one-stage visual grounding , author=. CVPR , year=

  62. [70]

    ICCV , year=

    Vision-language transformer and query generation for referring segmentation , author=. ICCV , year=

  63. [71]

    ECCV , year=

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection , author=. ECCV , year=

  64. [72]

    ECCV , year=

    Seqtr: A simple yet universal network for visual grounding , author=. ECCV , year=

  65. [73]

    ECCV , year=

    Mmbench: Is your multi-modal model an all-around player? , author=. ECCV , year=

  66. [74]

    Ray, Arijit and Radenovic, Filip and Dubey, Abhimanyu and Plummer, Bryan and Krishna, Ranjay and Saenko, Kate , editor =. Cola:. Advances in. 2023 , pages =

    2023

  67. [75]

    International Journal of Computer Vision , pages=

    Contextual object detection with multimodal large language models , author=. International Journal of Computer Vision , pages=. 2024 , publisher=

    2024

  68. [76]

    EMNLP , year=

    Referitgame: Referring to objects in photographs of natural scenes , author=. EMNLP , year=

  69. [77]

    EMNLP , year=

    Evaluating object hallucination in large vision-language models , author=. EMNLP , year=

  70. [78]

    arXiv preprint arXiv:2311.07397 , year=

    Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation , author=. arXiv preprint arXiv:2311.07397 , year=

    arXiv

  71. [79]

    EMNLP , year=

    Recantformer: Referring expression comprehension with varying numbers of targets , author=. EMNLP , year=

  72. [80]

    arXiv preprint arXiv:2403.18346 , year=

    Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective , author=. arXiv preprint arXiv:2403.18346 , year=

    arXiv

  73. [81]

    CVPR , year=

    Generation and comprehension of unambiguous object descriptions , author=. CVPR , year=

  74. [82]

    ACL , year=

    Words aren't enough, their order matters: On the robustness of grounding visual referring expressions , author=. ACL , year=

  75. [83]

    EMNLP , year=

    FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension , author=. EMNLP , year=

  76. [84]

    The Eleventh International Conference on Learning Representations , year=

    When and why vision-language models behave like bags-of-words, and what to do about it? , author=. The Eleventh International Conference on Learning Representations , year=

  77. [85]

    Yarom, Michal and Bitton, Yonatan and Changpinyo, Soravit and Aharoni, Roee and Herzig, Jonathan and Lang, Oran and Ofek, Eran and Szpektor, Idan , editor =. What. Advances in. 2023 , pages =

    2023

  78. [86]

    CVPR , year =

    Wang, Peng and Wu, Qi and Cao, Jiewei and Shen, Chunhua and Gao, Lianli and Hengel, Anton van den , title =. CVPR , year =

  79. [87]

    Findings of the Association for Computational Linguistics: ACL 2024 , year=

    CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples , author=. Findings of the Association for Computational Linguistics: ACL 2024 , year=

    2024

  80. [88]

    COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs , url =

    Le, Tiep and LAL, VASUDEV and Howard, Phillip , booktitle =. COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs , url =

  81. [89]

    arXiv preprint arXiv:2401.12168 , year =

    SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities , author =. arXiv preprint arXiv:2401.12168 , year =

    arXiv

  82. [90]

    ICCV , pages=

    Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition , author=. ICCV , pages=

  83. [91]

    NeurIPS , year=

    Described Object Detection: Liberating Object Detection with Flexible Expressions , author=. NeurIPS , year=

  84. [92]

    arXiv preprint arXiv:2310.01779 , year=

    Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption , author=. arXiv preprint arXiv:2310.01779 , year=

    arXiv

  85. [93]

    arXiv preprint arXiv:2403.20331 , year=

    Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models , author=. arXiv preprint arXiv:2403.20331 , year=

    arXiv

  86. [94]

    CVPR , pages=

    Cops-ref: A new dataset and task on compositional referring expression comprehension , author=. CVPR , pages=

  87. [95]

    CVPR , year=

    Graph-structured referring expression reasoning in the wild , author=. CVPR , year=

  88. [96]

    arXiv preprint arXiv:2310.11441 , year=

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v , author=. arXiv preprint arXiv:2310.11441 , year=

    arXiv

  89. [97]

    CVPR , year=

    Yolo-world: Real-time open-vocabulary object detection , author=. CVPR , year=

  90. [98]

    AAAI , year=

    ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection , author=. AAAI , year=

  91. [99]

    AAAI , year=

    How to evaluate the generalization of detection? a benchmark for comprehensive open-vocabulary detection , author=. AAAI , year=

  92. [100]

    AAAI , year=

    Hierarchical alignment-enhanced adaptive grounding network for generalized referring expression comprehension , author=. AAAI , year=

  93. [101]

    arXiv preprint arXiv:2407.07844 , year=

    OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion , author=. arXiv preprint arXiv:2407.07844 , year=

    arXiv

  94. [102]

    ACL , year=

    Groundinggpt: Language enhanced multi-modal grounding model , author=. ACL , year=

  95. [103]

    CVPR , year=

    Polyformer: Referring image segmentation as sequential polygon generation , author=. CVPR , year=

  96. [104]

    ECCV , year=

    End-to-end object detection with transformers , author=. ECCV , year=

  97. [105]

    arXiv preprint arXiv:2010.04159 , year=

    Deformable detr: Deformable transformers for end-to-end object detection , author=. arXiv preprint arXiv:2010.04159 , year=

    2010 arXiv

  98. [106]

    arXiv preprint arXiv:2203.03605 , year=

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection , author=. arXiv preprint arXiv:2203.03605 , year=

    arXiv

  99. [107]

    arXiv preprint arXiv:2401.02361 , year=

    An open and comprehensive pipeline for unified object grounding and detection , author=. arXiv preprint arXiv:2401.02361 , year=

    arXiv

  100. [108]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , year=

  101. [109]

    ICLR , year=

    Mllm can see? dynamic correction decoding for hallucination mitigation , author=. ICLR , year=

  102. [110]

    ICLR , year=

    Mitigating hallucination in large multi-modal models via robust instruction tuning , author=. ICLR , year=

  103. [111]

    arXiv preprint arXiv:2504.07615 , year=

    Vlm-r1: A stable and generalizable r1-style large vision-language model , author=. arXiv preprint arXiv:2504.07615 , year=

    arXiv

  104. [112]

    AAAI , year=

    Swift: a scalable lightweight infrastructure for fine-tuning , author=. AAAI , year=

  105. [113]

    and Stoica, Ion and Xing, Eric P

    Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P. , year =. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ url =

  106. [114]

    The Twelfth International Conference on Learning Representations , year=

    The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World , author=. The Twelfth International Conference on Learning Representations , year=

  107. [115]

    CVPR , year=

    Universal instance perception as object discovery and retrieval , author=. CVPR , year=

  108. [116]

    CVPR , year=

    Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection , author=. CVPR , year=

  109. [117]

    arXiv preprint arXiv:2312.02433 , year=

    Lenna: Language enhanced reasoning detection assistant , author=. arXiv preprint arXiv:2312.02433 , year=

    arXiv

  110. [118]

    arXiv preprint arXiv:2404.16821 , year=

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites , author=. arXiv preprint arXiv:2404.16821 , year=

    arXiv

  111. [119]

    NeurIPS , year=

    Cogvlm: Visual expert for pretrained language models , author=. NeurIPS , year=

  112. [120]

    arXiv preprint arXiv:2402.04236 , year=

    CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations , author=. arXiv preprint arXiv:2402.04236 , year=

    arXiv

  113. [121]

    arXiv preprint arXiv:2508.18265 , year=

    InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency , author=. arXiv preprint arXiv:2508.18265 , year=

    arXiv

  114. [122]

    arXiv preprint arXiv:2504.10479 , year=

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models , author=. arXiv preprint arXiv:2504.10479 , year=

    arXiv

  115. [123]

    arXiv preprint arXiv:2412.05271 , year=

    Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling , author=. arXiv preprint arXiv:2412.05271 , year=

    arXiv

  116. [124]

    Science China Information Sciences , year=

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites , author=. Science China Information Sciences , year=

  117. [125]

    CVPR , year=

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks , author=. CVPR , year=

  118. [126]

    arXiv preprint arXiv:2306.15195 , year=

    Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic , author=. arXiv preprint arXiv:2306.15195 , year=

    arXiv

  119. [127]

    NeurIPS , year=

    Direct preference optimization: Your language model is secretly a reward model , author=. NeurIPS , year=

  120. [128]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  121. [129]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  122. [130]

    arXiv preprint arXiv:2503.06749 , year=

    Vision-r1: Incentivizing reasoning capability in multimodal large language models , author=. arXiv preprint arXiv:2503.06749 , year=

    arXiv

  123. [131]

    ICCV , year=

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization , author=. ICCV , year=

  124. [132]

    NeurIPS , year=

    Learning to reason under off-policy guidance , author=. NeurIPS , year=

  125. [133]

    arXiv preprint arXiv:2502.01456 , year=

    Process reinforcement through implicit rewards , author=. arXiv preprint arXiv:2502.01456 , year=

    arXiv

  126. [134]

    arXiv preprint arXiv:2510.02173 , year=

    Learning to Reason for Hallucination Span Detection , author=. arXiv preprint arXiv:2510.02173 , year=

  127. [135]

    CVPR , year=

    Critic-v: Vlm critics help catch vlm errors in multimodal reasoning , author=. CVPR , year=

  128. [136]

    arXiv preprint arXiv:2505.14231 , year=

    Univg-r1: Reasoning guided universal visual grounding with reinforcement learning , author=. arXiv preprint arXiv:2505.14231 , year=

    arXiv

  129. [137]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 Technical Report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  130. [138]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  131. [139]

    ECCV , year=

    Modeling context in referring expressions , author=. ECCV , year=

  132. [140]

    and Xing, Eric and Yang, Ming-Hsuan and Khan, Fahad S

    Rasheed, Hanoona and Maaz, Muhammad and Shaji, Sahal and Shaker, Abdelrahman and Khan, Salman and Cholakkal, Hisham and Anwer, Rao M. and Xing, Eric and Yang, Ming-Hsuan and Khan, Fahad S. , title =. CVPR , year =

  133. [141]

    arXiv preprint arXiv:2312.02949 , year=

    Llava-grounding: Grounded visual chat with large multimodal models , author=. arXiv preprint arXiv:2312.02949 , year=

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.