Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Object referring should be a verifiable reasoning process over candidate instances, and training with CoT traces plus reinforcement learning delivers higher accuracy and far better abstention than direct box prediction.

desk verdict Useful dataset and solid in-domain gains, but the 'verifiable reasoning' claim rests on answer-conditioned GPT-4o rationalizations and is not actually demonstrated. read the letter →

arxiv 2506.04034 v1 pith:XCUMYSDH submitted 2025-06-04 cs.CV

Qing Jiang , Xingyu Chen , Zhaoyang Zeng , Junzhi Yu , Lei Zhang This is my paper · ORCID
classification cs.CV
keywords objectreferringexpressioncomprehensionchain-of-thoughtreasoningmultimodallargelanguagemodelsGRPOreinforcementlearninghallucinationrejectiongroundedvisualHumanRef-CoT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rex-Thinker argues that referring expression comprehension should be a verifiable reasoning process, not a direct bounding-box prediction: the model first receives candidate boxes for the referred category, then reasons about each candidate step by step before answering. To train this, the authors build HumanRef-CoT, 90,824 GPT-4o-generated traces in a planning–action–summarization format, and fine-tune a 7B multimodal LLM with supervised cold start followed by GRPO reinforcement learning. If the results hold, object referring can be both more accurate and much better at rejecting expressions that match nothing in the image, with reasoning that a human can inspect.

What carries the argument

The load-bearing object is the structured CoT trace with three stages: planning (decompose the expression into subgoals), action (evaluate each candidate box one by one with visual checks), and summarization (aggregate and re-check). Candidate boxes come from an open-vocabulary detector, Grounding DINO, and are fed as box hints; the model's final answer must be JSON bboxes that exactly match one of the provided hints, which is enforced by an F1 reward computed at IoU=1 during GRPO training. GRPO (Group Relative Policy Optimization) is the reinforcement-learning update that samples a group of responses, normalizes their rewards, and reinforces the better ones while penalizing KL divergence from the SFT model.

What would settle it

Re-generate HumanRef-CoT with GPT-4o prompts that omit the green/red answer marks and the stated correct answer (or use only red marks), train the identical pipeline, and measure DF1 and rejection on HumanRef. If performance is preserved, the traces teach real visual reasoning; if it drops substantially, the traces leaked the answer. A complementary check: run the trained model on images where the supplied hints are deliberately wrong and count how often reasoning corrects them.

Watch

Extended reading notes

Core claim

The paper's central claim is that turning object referring into an explicit chain-of-thought task over candidate instances makes predictions verifiable and trustworthy without sacrificing accuracy. On HumanRef, Rex-Thinker-GRPO achieves an average DensityF1 of 83.5 and a rejection score of 68.2, beating the retrieval-based RexSeek-7B baseline (82.3 DF1, 54.1 rejection) and every direct-prediction or retrieval baseline tested. The same model, trained only on HumanRef-CoT, reaches 83.3 accuracy on RefCOCOg zero-shot, and 88.8 after GRPO fine-tuning on that dataset; it also corrects a deliberately wrong hint label during reasoning. The authors interpret these results as evidence that step-by-step grounded reasoning, not just candidate retrieval, is what drives both accuracy and abstention.

Load-bearing premise

The paper's results depend on the GPT-4o-generated reasoning traces being genuine visual reasoning rather than paraphrases of the supplied ground-truth answers and green/red marks.

Editorial extensions

If this is right

  • Referring models can be made explainable: every prediction is traceable to a specific candidate box and a specific reasoning step.
  • The ability to abstain is learnable and improves with CoT: rejection score rises to 68.2 on HumanRef, well above the 54.1 of the prior best retrieval baseline.
  • A model trained only on human-centric referring can generalize zero-shot to other categories, reaching 83.3 accuracy on RefCOCOg and handling wrong hint labels via reasoning.
  • Two-stage training matters: CoT cold start before GRPO raises final DensityF1 from 77.8 to 83.5 and keeps reasoning structured rather than incoherent.
  • Providing candidate boxes is a major driver of accuracy: adding box hints improves average DensityF1 by 10.8 points in the ablation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the same planning–action–summarization loop could be applied to other detection-style grounding tasks, such as counting or visual question answering with abstention, since the method does not depend on person-specific priors beyond the training data.
  • A cleaner isolation of the CoT contribution would hold the retrieval interface fixed and vary only the reasoning supervision; the reported gap between Rex-Thinker-Plain and Rex-Thinker-CoT (53.5 vs 67.3 rejection) already suggests the CoT is the active ingredient.
  • A natural stress test is to generate CoT traces without green/red visual marks or supplied answers and see whether the trained model still reasons correctly; the appendix's GPT-4o-alone score of 53.2 DF1 suggests this would be a strong test of data quality.
  • The paper leaves open a consistency reward that would force the final answer count to match the reasoning summary, which it identifies as future work; adding such a reward would likely reduce the observed reasoning–answer mismatches.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Rex-Thinker, a multimodal large language model that reformulates referring expression comprehension as a chain-of-thought reasoning task. Given an image, a referring expression, and candidate object boxes from an open-vocabulary detector, the model produces a structured planning–action–summarization trace and then selects the matching boxes. To support this, the authors introduce HumanRef-CoT, a dataset of 90,824 GPT-4o-generated reasoning traces built on the HumanRef benchmark, and train Rex-Thinker in two stages: cold-start supervised fine-tuning on the traces followed by GRPO reinforcement learning with an F1-plus-format reward. In-domain experiments on HumanRef report improved DensityF1 and a large gain in rejection score over prior work, and zero-shot experiments on RefCOCOg show competitive accuracy relative to models not trained on HumanRef. The paper's central claim is that the CoT formulation makes referring 'verifiable' and 'trustworthy', grounding each reasoning step in a candidate region.

Significance. If the central claims are established, this is a valuable contribution: it provides a large CoT-style dataset for referring, demonstrates a two-stage SFT+RL recipe that improves detection accuracy and rejection behavior, and offers a template for interpretable referring via candidate evaluation. The dataset and code are likely to be reused by the community. However, the strongest claims—state-of-the-art performance on HumanRef and the 'grounded'/'verifiable' nature of the reasoning—rest on assumptions that the current experiments do not fully test. The answer-conditioned data engine and the lack of a fair baseline comparison are the main risks to the paper's headline conclusions.

major comments (4)
  1. [Section 3.2.2 / Appendix Figure 6 / Appendix Table 7] The GPT-4o annotation pipeline is explicitly answer-conditioned: the prompt includes the ground-truth answer in METAINFO and marks the target boxes in green, and the system prompt instructs the model to 'pretend that you are thinking about it yourself.' This design makes the resulting CoT traces post-hoc rationalizations of a supplied answer rather than independent perceptual reasoning. The load-bearing concern is that the trained model may imitate the narrative structure of these traces without learning to evaluate candidates from visual evidence alone. Appendix Table 7 shows GPT-4o without answer hints achieves only 53.2 DF1 and 14.8 rejection score, so the teacher cannot perform the task itself; the traces thus carry information beyond what the model will have at inference. The paper's claim that Rex-Thinker is 'grounded' and 'verifiable' (Section 1) requires that the model's reasoning is faithful to visual evidence, and this is not established. The reward design (Section 4.2.3) includes only F1 and format rewards, and Appendix A.3.2 documents cases where reasoning and answer disagree. To support the central claim, the authors should either provide evidence of causal grounding (e.g., perturbing visual regions and showing the reasoning changes accordingly), or train a model on traces generated without answer hints (or with a teacher that can actually perform the task) and show the performance and faithfulness are preserved. Without such evidence, the 'grounded' property remains unsupported.
  2. [Section 5.2 / Table 2] The claim of state-of-the-art performance on HumanRef is not supported by a fair comparison. All baselines in Table 2 (except RexSeek-7B) are presumably general-purpose referring models that were not trained on the HumanRef training set, whereas Rex-Thinker is trained on HumanRef-CoT, which is derived from HumanRef. The in-domain evaluation therefore measures transfer for the baselines and training-set performance for Rex-Thinker. The paper should report, at minimum, results for the same baselines after supervised fine-tuning on the HumanRef training set, or clearly state which baselines were trained on HumanRef. The 1.2-point DF1 improvement over RexSeek-7B is also within the range where seed variability could matter, so the SOTA claim needs error bars or significance testing.
  3. [Section 5.2 / Tables 2 and 3] All results are reported from a single training run. The key differences (e.g., Rex-Thinker-GRPO vs. Rex-Thinker-CoT DF1 83.5 vs 82.3; rejection 68.2 vs 67.3) are small, and GRPO training is stochastic. The paper should report mean ± std over at least 3 seeds, or a paired test, for the main comparisons, especially the rejection score improvement over RexSeek-7B (54.1 vs 68.2). Without this, the claimed improvements may not be statistically reliable.
  4. [Section 5.3 / Abstract] The statement that the approach 'outperforms standard baselines in both precision and interpretability on in-domain evaluation, while also demonstrating improved ability to reject hallucinated outputs and strong generalization in out-of-domain settings' is misleading when read alongside Table 3. On RefCOCOg, the zero-shot Rex-Thinker-GRPO (83.3) is below all listed baselines, e.g., ChatRex-7B (89.8) and Qwen2.5-VL-7B (87.2). The 'strong generalization' claim is only meaningful if framed as zero-shot capability relative to a model trained on HumanRef only, not as beating supervised baselines. The paper should explicitly separate the in-domain SOTA claim (which itself needs a fair comparison) from the out-of-domain generalization claim, and avoid the impression that the model is SOTA on RefCOCOg.
minor comments (5)
  1. [Table 2 caption] The word 'blod' in the caption should be 'bold'.
  2. [Throughout] The model name appears inconsistently as 'Rex-Thinker' and 'RexThinker' (e.g., Figures 5 and 8, Section 6). Please standardize.
  3. [Section 5.4 / Appendix A.1.2] Table 4 in Section 5.4 and Table 6 in Appendix A.1.2 appear to duplicate the same ablation study with the same title; if they are the same, keep one reference.
  4. [Section 5.3 / Figure 4] The claim that Rex-Thinker 'rectifies the error' when given an incorrect hint label is based on a single qualitative example (Figure 4). A quantitative measure of self-correction would strengthen this claim.
  5. [Section 3.2] The paper would benefit from a table describing the HumanRef-CoT dataset statistics (number of images, mean candidates, distribution across the six subsets) and how the test split of HumanRef relates to the CoT annotation set.

Circularity Check

1 steps flagged · score 4.0 of 10

Answer-conditioned GPT-4o traces make the 'grounded reasoning' claim partially self-definitional; the held-out box-level scores remain independent empirical results.

  1. self definitional [Section 3.2.2 (GPT-4o Annotation) and Section 4.2.1 (SFT Cold Start)]
    "In essence, we provide GPT-4o with both the referring expression and its ground-truth answer, and prompt it to generate step-by-step reasoning in our CoT format. To ensure annotation quality, we retain only examples where GPT-4o’s final prediction matches the ground-truth label."

    The 'grounded, verifiable' reasoning that the paper presents as the model's learned capability is generated by GPT-4o after the ground-truth answer and green/red marks were provided, with the system prompt telling the model to 'pretend that you are thinking about it yourself.' SFT then applies token-level cross-entropy to both the reasoning trace and the final answer, so the answer-consistent trace is a training target, not an independently derived visual inference. Appendix Table 7 shows GPT-4o without answer hints only reaches 53.2 DF1, confirming the answer-conditioned traces carry information unavailable at inference.

full rationale

The paper's central contribution is an empirical system: build the HumanRef-CoT dataset (Sec 3.2), train with SFT and GRPO (Sec 4.2), and evaluate on HumanRef and RefCOCOg (Sec 5). For the quantitative claims—DF1, rejection score, and zero-shot RefCOCOg accuracy—the derivation chain is not circular: the model is evaluated on held-out benchmarks and compared against Rex-Thinker-Plain and external baselines. The self-citations to HumanRef [21] and RexSeek-7B [21] are normal and are not used as a uniqueness theorem or to forbid alternatives. The one genuinely self-definitional element is the 'grounded/verifiable' interpretation: the reasoning traces used as supervision were generated by GPT-4o with the ground-truth answer and green/red markers supplied, and SFT trains the model on those exact traces, so the model's ability to emit answer-consistent reasoning is partly a reconstruction of the annotation rather than independent evidence of visual grounding. Appendix A.3.2's documented reasoning-answer mismatches further weaken the groundedness claim. This does not invalidate the box-level accuracy results; it only means the interpretability/faithfulness claim should not be read as independently established by the paper. Hence score 4 rather than higher.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or formal entities. The free parameters are training hyperparameters and the IoU matching criterion, all chosen by hand. The main tacit assumptions are the completeness of HumanRef boxes, the quality of GPT-4o reasoning traces, and the recall of Grounding DINO for out-of-domain candidates.

free parameters (3)
  • reward weight lambda = 0.9
    Chosen by hand in Section 4.2.3 to weight F1 reward versus format reward in the GRPO objective.
  • KL penalty beta = 0.04
    Standard GRPO hyperparameter set in Section 5.1; affects the strength of the KL penalty toward the reference model.
  • IoU matching threshold = 1 (exact overlap)
    Section 4.2.3 defines a match only when a predicted box exactly overlaps with a ground-truth box among the provided hints. This design choice shapes both reward and evaluation.
assumptions (5)
  • domain assumption HumanRef provides complete person boxes and ground-truth labels for all images.
    Used to generate CoT annotations and to define box hints in training and in-domain evaluation. If boxes are incomplete, candidate coverage fails.
  • domain assumption GPT-4o's CoT annotations are high-quality and learnable by Qwen2.5-VL.
    The entire HumanRef-CoT dataset depends on GPT-4o outputs, filtered only by final answer match, with no human validation of reasoning traces.
  • domain assumption Grounding DINO returns all relevant candidates in out-of-domain evaluation.
    Rex-Thinker can only select from provided box hints, so any missed object is unrecoverable (Section 5.3).
  • ad hoc to paper Exact IoU=1 matching is a valid reward and evaluation signal.
    This encourages selection from hints but may penalize slightly shifted detections that are semantically correct.
  • domain assumption Two-stage SFT and GRPO training improves accuracy and generalization.
    The paper assumes that reward-driven exploration after supervised cold-start yields better reasoning paths, an assumption supported only by the reported tables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning." pith.science (2026). https://pith.science/paper/XCUMYSDH

@misc{pith2026250604034,
  author       = {Pith},
  title        = {Pith review of: Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XCUMYSDH}},
  note         = {Machine review of arXiv:2506.04034}
}
read the original abstract

Object referring aims to detect all objects in an image that match a given natural language description. We argue that a robust object referring model should be grounded, meaning its predictions should be both explainable and faithful to the visual content. Specifically, it should satisfy two key properties: 1) Verifiable, by producing interpretable reasoning that justifies its predictions and clearly links them to visual evidence; and 2) Trustworthy, by learning to abstain when no object in the image satisfies the given expression. However, most methods treat referring as a direct bounding box prediction task, offering limited interpretability and struggling to reject expressions with no matching object. In this work, we propose Rex-Thinker, a model that formulates object referring as an explicit CoT reasoning task. Given a referring expression, we first identify all candidate object instances corresponding to the referred object category. Rex-Thinker then performs step-by-step reasoning over each candidate to assess whether it matches the given expression, before making a final prediction. To support this paradigm, we construct a large-scale CoT-style referring dataset named HumanRef-CoT by prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a structured planning, action, and summarization format, enabling the model to learn decomposed, interpretable reasoning over object candidates. We then train Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach the model how to perform structured reasoning, followed by GRPO-based RL learning to improve accuracy and generalization. Experiments show that our approach outperforms standard baselines in both precision and interpretability on in-domain evaluation, while also demonstrating improved ability to reject hallucinated outputs and strong generalization in out-of-domain settings.

Figures

Figures reproduced from arXiv: 2506.04034 by the authors.

Figure 1
Figure 1. An example of Rex-Thinker for object referring with CoT reasoning of planning (task [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed CoT reasoning referring data engine. We prompt GPT-4o to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the Rex-Thinker architecture and our two-stage training methods [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (43 more)
Figure 4
Figure 4. Figure 4: The out-of-domain result. We use Rex-Thinker-GPRO trained on HumanRef-CoT to infer [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Predictions from a model that was trained with GRPO [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The system prompt used to instruct GPT-4o on visual reasoning for HumanRef-CoT. It [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: GRPO training curves showing accuracy reward, format reward, and completion length [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Example of reasoning–answer mismatch. The number of predicted objects differs between [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Attribute referring example. CoT reasoning improves interpretability, ensuring its consistency and robustness remains an open challenge. Future work should address these risks to ensure safe and responsible deployment. A.4 Visualization Results In this section, we pres…
Figure 10
Figure 10. Figure 10: Attribute referring example. Question: Please detect apple with disease, and tell me what kind of disease it may be.. Answer: <think>OK, the user needs us to detect the apple with disease and tell what kind of disease it may be. In order to accomplish this task, I nee…
Figure 11
Figure 11. Figure 11: Attribute referring example. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Attribute referring example. Question: Please detect damaged container.. Answer: <think>OK, the user needs us to detect the damaged container. In order to accomplish this task, I need to break it down into the following steps: - Step 1: Examine each of the 31 containe…
Figure 13
Figure 13. Figure 13: Attribute referring example. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Attribute referring example. Question: Please detect the two people holding hands. Answer: <think>OK, the user needs us to detect two people holding hands. To accomplish this task, I need to break it down into the following steps: - Step 1: Identify pairs of people en…
Figure 15
Figure 15. Figure 15: Interaction referring example. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Interaction referring example. Question: Please detect all people with something in their hands. Answer: <think>OK, the user needs us to detect all people with something in their hands. To accomplish this task, I need to break it down into the following steps: - Step …
Figure 17
Figure 17. Figure 17: Interaction referring example. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Interaction referring example. Question: Please detect the leader. Answer: <think>OK, the user needs us to detect the leader. In order to accomplish this task, I need to break it down into the following steps: - Step 1: Observe the positions of all 18 cars on the trac…
Figure 19
Figure 19. Figure 19: Position referring example. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Position referring example. Question: Please detect Hot dogs on the grill Answer: <think> OK, the user needs us to detect the hot dogs on the grill. In order to accomplish this task, I need to break it down into the following steps: - Step 1: Determine the position of…
Figure 21
Figure 21. Figure 21: Position referring example. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]
Figure 22
Figure 22. Figure 22: Celebrity referring example. Question: Please detect all non-real persons. Answer: <think>OK, the user needs us to detect all non-real persons. In order to accomplish this task, I need to break it down into the following steps: - Step 1: Determine which people in the …
Figure 23
Figure 23. Figure 23: Reasoning referring example. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]
Figure 24
Figure 24. Figure 24: Reasoning referring example. Question: Please detect the dog sleep on the bed with a pot under its body Answer: <think> OK, the user needs us to detect the dog sleep on the bed with a pot under its body. To accomplish this task, I need to break it down into the follow…
Figure 25
Figure 25. Figure 25: Rejection referring example. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_25.png]
Figure 26
Figure 26. Figure 26: Rejection referring example. Question: Please detect Kobe Bryant. Answer: <think>OK, the user needs us to detect Kobe Bryant. To accomplish this task, I need to break it down into the following steps: - Step 1: Identify each person based on their appearance and basket…
Figure 27
Figure 27. Figure 27: Rejection referring example. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_27.png]
Figure 28
Figure 28. Figure 28: Reasoning referring example with multi-task chat. [PITH_FULL_IMAGE:figures/full_fig_p028_28.png]
Figure 29
Figure 29. Figure 29: In-context prompt for attribute subset in HumanRef-CoT. (analysis) OK, the user needs us to detect the person with brownish - black curly hair. In order to accomplish this task, I need to break it down into the following steps: - Step 1: Examine the hair color and tex…
Figure 30
Figure 30. Figure 30: Visualization of GPT-4o’s output on the attribute subset. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_30.png]
Figure 31
Figure 31. Figure 31: In-context prompt for position (inner) subset in HumanRef-CoT. (analysis) OK, the user needs us to detect the second person wearing gloves when counting from right to left. To accomplish this task, I need to break it down into the following steps: - Step 1: Determine …
Figure 32
Figure 32. Figure 32: Visualization of GPT-4o’s output on the position (inner) subset. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_32.png]
Figure 33
Figure 33. Figure 33: In-context prompt for position (outer) subset in HumanRef-CoT. Example 1 : ``` METAINFO: { “total_num_of_person”: 5, “answer”: “person 1”, “order_from_left_to_right”: [person 1, person 4, person 2, person 5, person 3] } Question: the person to the left of the green li…
Figure 34
Figure 34. Figure 34: Visualization of GPT-4o’s output on the position (outer) subset. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_34.png]
Figure 35
Figure 35. Figure 35: In-context prompt for interaction (inner) subset in HumanRef-CoT. (analysis) OK, the user needs us to detect two people who are holding hands. To accomplish this task, I need to break it down into the following steps: - Step 1: Identify the pair of people engaged in a…
Figure 36
Figure 36. Figure 36: Visualization of GPT-4o’s output on the interaction (inner) subset. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_36.png]
Figure 37
Figure 37. Figure 37: In-context prompt for interaction (outer) subset in HumanRef-CoT. (analysis) OK, the user needs us to find the person holding a microphone in the hand. To accomplish this task, I need to break it down into the following steps: - Step 1: Check the hands of each person …
Figure 38
Figure 38. Figure 38: Visualization of GPT-4o’s output on the interaction (outer) subset. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_38.png]
Figure 39
Figure 39. Figure 39: In-context prompt for reasoning (inner position) subset in HumanRef-CoT. (analysis) OK, the user needs us to detect the woman to the right of the man wearing a gold tie. In order to accomplish this task, I need to break it down into the following steps: Step 1: Find t…
Figure 40
Figure 40. Figure 40: Visualization of GPT-4o’s output on the reasoning (inner position) subset. . 34 [PITH_FULL_IMAGE:figures/full_fig_p034_40.png]
Figure 41
Figure 41. Figure 41: In-context prompt for reasoning (attribute) subset in HumanRef-CoT. (analysis) OK, the user needs us to detect the person straddling someone's shoulders but not wearing long blue ripped jeans. To accomplish this task, I need to break it down into the following steps: …
Figure 42
Figure 42. Figure 42: Visualization of GPT-4o’s output on the reasoning (attribute) subset.. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_42.png]
Figure 43
Figure 43. Figure 43: In-context prompt for celebrity recognition subset in HumanRef-CoT. (analysis) OK, the user needs us to detect Donald Sutherland. To accomplish this task, I need to break it down into the following steps: - Step 1: Recall what Donald Sutherland looks like and match hi…
Figure 44
Figure 44. Figure 44: Visualization of GPT-4o’s output on the celebrity recognition subset.. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_44.png]
Figure 45
Figure 45. Figure 45: In-context prompt for rejection subset in HumanRef-CoT. (analysis) OK, the user needs us to detect the person wearing sunglasses. To accomplish this task, I need to break it down into the following steps: - Step 1: Examine the attire and accessories of each person to …
Figure 46
Figure 46. Figure 46: Visualization of GPT-4o’s output on the rejection subset.. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_46.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A unified 210K medical grounding dataset and full fine-tuning raise a general model's localization F1 from 10.6 to 85.6 on held-out data and improve cross-source transfer.

  2. Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation

    cs.CV 2025-06 conditional novelty 7.0 of 10

    BadSem shows that semantic mismatches between images and text can serve as stealthy backdoor triggers for VLMs, achieving near-perfect attack success with low poisoning rates.

  3. RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

    cs.CV 2025-12 conditional novelty 6.0 of 10

    RefBench-PRO organizes REC into attribute, position, interaction, relation, commonsense, and reject tasks; no tested MLLM exceeds 72%, and Ref-R1 raises Qwen2.5-VL-7B from 57.6 to 69.4 on it.

  4. VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

    cs.CV 2026-07 reject novelty 4.0 of 10

    VocaDet detects arbitrary objects by retrieving multi-granularity visual tokens from a sample-built vector database of position-debiased DINOv3 features and topology, without detector training.

Reference graph

Works this paper leans on

76 extracted references · 22 canonical work pages · cited by 4 Pith papers

  1. [1]

    Pixtral 12b.arXiv preprint arXiv:2410.07073, 2024

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b.arXiv preprint arXiv:2410.07073, 2024

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 35:23716–23736, 2022

    2022

  3. [3]

    Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    arXiv 2025

  5. [5]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

  6. [6]

    Pix2seq: A language modeling framework for object detection.arXiv preprint arXiv:2109.10852, 2021

    Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection.arXiv preprint arXiv:2109.10852, 2021

    arXiv 2021

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jia...

    2025

Show all 76 references
  1. [9]

    Yolo-world: Real- time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024

    2024

  2. [10]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025

    2025 arXiv

  3. [11]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  4. [12]

    Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.arXiv preprint arXiv:2503.17352, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.arXiv preprint arXiv:2503.17352, 2025

    2025 arXiv

  5. [13]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  6. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 10

    2025 arXiv

  7. [15]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    2025 arXiv

  8. [16]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [17]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  10. [18]

    T-rex: Counting by visual prompting.arXiv preprint arXiv:2311.13596, 2023

    Qing Jiang, Feng Li, Tianhe Ren, Shilong Liu, Zhaoyang Zeng, Kent Yu, and Lei Zhang. T-rex: Counting by visual prompting.arXiv preprint arXiv:2311.13596, 2023

    2023 arXiv

  11. [19]

    T-rex2: Towards generic object detection via text-visual prompt synergy

    Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. InEuropean Conference on Computer Vision, pages 38–57. Springer, 2024

    2024

  12. [20]

    T-rex2: Towards generic object detection via text-visual prompt synergy

    Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. InEuropean Conference on Computer Vision, pages 38–57. Springer, 2025

    2025

  13. [21]

    Referring to any person, 2025

    Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Qin Liu, and Lei Zhang. Referring to any person, 2025

    2025

  14. [22]

    Chatrex: Taming multimodal llm for joint perception and understanding.arXiv preprint arXiv:2411.18363, 2024

    Qing Jiang, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, Lei Zhang, et al. Chatrex: Taming multimodal llm for joint perception and understanding.arXiv preprint arXiv:2411.18363, 2024

    2024 arXiv

  15. [23]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  16. [24]

    Aria: An open multimodal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model.arXiv preprint arXiv:2410.05993, 2024

    2024 arXiv

  17. [25]

    Visual in-context prompting

    Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jian- wei Yang, Chunyuan Li, et al. Visual in-context prompting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12861–12871, 2024

    2024

  18. [26]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  19. [27]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2022

  20. [28]

    Eagle 2: Building post-training data strategies from scratch for frontier vision-language models.arXiv preprint arXiv:2501.14818, 2025

    Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-language models.arXiv preprint arXiv:2501.14818, 2025

    2025 arXiv

  21. [29]

    A real-time cross-modality correlation filtering method for referring expression comprehension

    Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross-modality correlation filtering method for referring expression comprehension. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10880–10889, 2020

    2020

  22. [30]

    Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025

    Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng. Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025

    2025 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  24. [32]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  25. [33]

    Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025. 11

    2025 arXiv

  26. [34]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  27. [35]

    Multi- task collaborative network for joint referring expression comprehension and segmentation

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi- task collaborative network for joint referring expression comprehension and segmentation. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages...

    2020

  28. [36]

    Groma: Localized visual tokenization for grounding multimodal large language models.arXiv preprint arXiv:2404.13013, 2024

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models.arXiv preprint arXiv:2404.13013, 2024

    2024 arXiv

  29. [37]

    Deepperception: Advancing r1-like cognitive visual perception in mllms for knowledge- intensive visual grounding.arXiv preprint arXiv:2503.12797, 2025

    Xinyu Ma, Ziyang Ding, Zhicong Luo, Chi Chen, Zonghao Guo, Derek F Wong, Xiaoyi Feng, and Maosong Sun. Deepperception: Advancing r1-like cognitive visual perception in mllms for knowledge- intensive visual grounding.arXiv preprint arXiv:2503.12797, 2025

    2025

  30. [38]

    Yuille, and Kevin Murphy

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. InCVPR, pages 11–20, 2016

    2016

  31. [39]

    MM1: methods, analysis & insights from multimodal LLM pre-training.arXiv: 2403.09611, 2024

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, J...

    2024 arXiv

  32. [40]

    Simple open- vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. InEuropean conference on computer vision, pages 728–755....

    2022

  33. [41]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025

    2025

  34. [42]

    Competitive programming with large reasoning models, 2025

    OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Do- han, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contr...

    2025

  35. [43]

    Gpt-4v(ision) system card.https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023

    OpenAI. Gpt-4v(ision) system card.https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023

    2023

  36. [44]

    Skywork r1v: Pioneering multimodal reasoning with chain-of-thought.arXiv preprint arXiv:2504.05599, 2025

    Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, et al. Skywork r1v: Pioneering multimodal reasoning with chain-of-thought.arXiv preprint arXiv:2504.05599, 2025

    2025 arXiv

  37. [45]

    Referring expression comprehension: A survey of methods and datasets.IEEE Transactions on Multimedia, 23:4426–4440, 2020

    Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring expression comprehension: A survey of methods and datasets.IEEE Transactions on Multimedia, 23:4426–4440, 2020

    2020

  38. [46]

    Dino-x: A unified vision model for open-world object detection and understanding.arXiv preprint arXiv:2411.14347, 2024

    Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, et al. Dino-x: A unified vision model for open-world object detection and understanding.arXiv preprint arXiv:2411.14347, 2024

    2024 arXiv

  39. [47]

    Grounding dino 1.5: Advance the" edge" of open-set object detec- tion.arXiv preprint arXiv:2405.10300, 2024

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detec- tion.arXiv preprint arXiv:2405.10300, 2024

    2024 arXiv

  40. [48]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  41. [49]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  42. [50]

    Paligemma 2: A family of versatile vlms for transfer.arXiv preprint arXiv:2412.03555, 2024

    Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: A family of versatile vlms for transfer.arXiv preprint arXiv:2412.03555, 2024. 12

    2024 arXiv

  43. [51]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  44. [52]

    Llamav-o1: Rethinking step-by-step visual reasoning in llms.arXiv preprint arXiv:2501.06186, 2025

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms.arXiv preprint arXiv:2501.06186, 2025

    2025 arXiv

  45. [53]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  46. [54]

    Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  47. [55]

    Skywork r1v2: Multimodal hybrid reinforcement learning for reason- ing.arXiv preprint arXiv:2504.16656, 2025

    Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, et al. Skywork r1v2: Multimodal hybrid reinforcement learning for reason- ing.arXiv preprint arXiv:2504.16656, 2025

    2025 arXiv

  48. [56]

    Aligning bag of regions for open-vocabulary object detection

    Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15254–15264, 2023

    2023

  49. [57]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  50. [58]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-thought, 2025

    Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. Towards system 2 reasoning in llms: Learning how to think with m...

    2025

  51. [59]

    Self-rewarding correction for mathematical reasoning, 2025

    Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning, 2025

    2025

  52. [60]

    Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  53. [61]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  54. [62]

    Kptllm: Unveiling the power of large language model for keypoint comprehension

    Jie Yang, ZENG Wang, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, and Ruimao Zhang. Kptllm: Unveiling the power of large language model for keypoint comprehension. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2025

    2025

  55. [63]

    Dynamic graph attention for referring expression comprehension

    Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression comprehension. InProceedings of the IEEE/CVF international conference on computer vision, pages 4644–4653, 2019

    2019

  56. [64]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  57. [65]

    Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.Advances in Neural Information Processing Systems, 35:9125–9138, 2022

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.Advances in Neural Information Processing Systems, 35:9125–9138, 2022

    2022

  58. [66]

    Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih- Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    2023 arXiv

  59. [67]

    Perception-r1: Pioneering perception policy with reinforcement learning

    En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-r1: Pioneering perception policy with reinforcement learning. arXiv preprint arXiv:2504.07954, 2025

    2025 arXiv

  60. [68]

    Mattnet: Modular attention network for referring expression comprehension

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. InProceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 1307–1315, 2018. 13

    2018

  61. [69]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. InECCV, volume 9906, pages 69–85, 2016

    2016

  62. [70]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14393–14402, 2021

    2021

  63. [71]

    Griffon: Spelling out all object locations at any granularity with large language models

    Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. InEuropean Conference on Computer Vision, pages 405–422. Springer, 2025

    2025

  64. [72]

    Griffon v2: Advanc- ing multimodal perception with high-resolution scaling and visual-language co-referring.arXiv preprint arXiv:2403.09333, 2024

    Yufei Zhan, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon v2: Advanc- ing multimodal perception with high-resolution scaling and visual-language co-referring.arXiv preprint arXiv:2403.09333, 2024

    2024 arXiv

  65. [73]

    Referring expression comprehension with semantic visual relationship and word mapping

    Chao Zhang, Weiming Li, Wanli Ouyang, Qiang Wang, Woo-Shik Kim, and Sunghoon Hong. Referring expression comprehension with semantic visual relationship and word mapping. InProceedings of the 27th ACM International Conference on Multimedia, pages 1258–1266, 2019

    2019

  66. [74]

    Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    2024 arXiv

  67. [75]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  68. [76]

    the third man to the left of the boy wearing a red shirt

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 14 A Appendix A.1 M...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.