Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Region-based Cluster Discrimination for Visual Representation Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RICE, a region-aware cluster discrimination pretraining method, produces a vision encoder that outperforms previous encoders on segmentation, dense detection, and OCR-heavy multimodal perception, including surpassing a specialized OCR…

desk verdict Solid region-level cluster discrimination paper with a real overclaim in the abstract; worth reviewing but needs a tone-down and variance reporting. read the letter →

arxiv 2507.20025 v1 pith:WSYMD6LD submitted 2025-07-26 cs.CV

Yin Xie , Kaicheng Yang , Xiang An , Kun Wu , Yongle Zhao , Weimo Deng , Zimin Ran , Yumeng Wang
show 4 more authors
Ziyong Feng Roy Miles Ismail Elezi Jiankang Deng
This is my paper
classification cs.CV
keywords visualrepresentationlearningregion-basedpretrainingclusterdiscriminationOCRdensepredictionsegmentationmultimodalLLMvisiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a vision encoder pretrained with region-aware cluster discrimination—supervising region-level object recognition and OCR through a single classification loss—can outperform global-contrastive encoders such as CLIP and SigLIP on dense prediction, OCR, segmentation, and multimodal perception tasks. The authors build RICE, which extracts region embeddings through a Region Transformer layer that masks attention to each region, and trains them against one million semantic cluster centers for objects and token embeddings for OCR text. On benchmarks such as DocVQA, COCO detection, referring segmentation, and video tracking, RICE consistently beats the baselines, and at 560px resolution it surpasses the specialized backbone of a leading multimodal model on document question answering. If the results hold, a single pretrained encoder could provide object-level semantics and OCR reading in a single forward pass, reducing the need for separate specialized vision backbones.

What carries the argument

The central mechanism is the unified region cluster discrimination loss, which turns every image region into a classification problem: object regions are classified against one million k-means cluster centers derived from frozen CLIP features (single-label), and OCR regions are classified against token embeddings of the detected text (multi-label). The Region Transformer layer carries out this supervision: it applies a region-specific visibility mask (0 inside a region, −∞ outside) to a standard multi-head attention operation, so each region's class token only sees tokens within that region, and produces fixed-length embeddings for all regions in one forward pass. Training uses a margin-based softmax with L2-normalized features and centers, and a random negative-sampling strategy that selects 10% of class centers as negatives to avoid conflicting gradients.

What would settle it

Train a variant of RICE where the object region labels are replaced by random assignments to the one million centers, keeping the region masking and OCR losses identical. If the variant still matches RICE's gains on COCO detection and DocVQA, then the specific object cluster targets are not what drives the improvement.

Watch

Extended reading notes

Core claim

RICE is a fully region-supervised visual representation learning method. The paper's central claim is that treating image regions as classification targets—single-label object clusters plus multi-label OCR token embeddings—under one unified cluster discrimination loss produces a vision transformer whose local features transfer to a broad set of dense and OCR-heavy tasks better than global vision-language contrastive models and previous cluster discrimination models. The method constructs a dataset of 400 million images with 2 billion object regions (from segmentation masks) and 400 million OCR regions (from detected text), assigns each object region a pseudo-label by nearest-neighbor matching of its CLIP feature to one of one million k-means centers, and then trains a ViT with additional Region Transformer layers that restrict self-attention to region tokens. The object and OCR losses share the same softmax-margin formulation, enabling distributed training at a global batch size of 32K on 64 GPUs. The paper reports that this encoder outperforms CLIP, SigLIP, MLCD, and others on COCO/LVIS detection and segmentation, referring segmentation, DocVQA/OCRBench, and MLLM visual perception, reaching 87.38% DocVQA at 560px in a standard multimodal LLM setup and beating a specialized OCR-optimized vision backbone.

Load-bearing premise

The object region pseudo-labels come from k-means on frozen CLIP features, so RICE's object supervision is only as good as CLIP's ability to organize region features into semantically meaningful clusters; if those clusters are noisy or biased, the model inherits that noise.

Editorial extensions

If this is right

  • A single vision encoder from this method can serve as the vision tower for multimodal LLMs, improving OCR and dense understanding without a separate OCR-optimized backbone.
  • Region-level supervision transfers to object tracking: RICE improves video tracking success rates on LaSOT, TrackingNet, GOT-10k, and TNL2K over prior encoders.
  • The unified object/OCR classification loss scales to billion-scale region data, suggesting that region-level cluster discrimination is a practical path to further scaling visual representation learning.
  • Because the region attention layer produces fixed-length region embeddings in one forward pass, downstream tasks like referring segmentation can directly consume region features without region-specific fine-tuning of the encoder.
  • The ablation results indicate that 10 sampled regions per image, 1–2 million cluster centers, and a 0.1 negative sampling ratio are near-optimal; these are actionable defaults for follow-up work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains persist when the frozen CLIP feature extractor is replaced by a different (e.g., SigLIP-derived) feature backbone for clustering, region cluster discrimination could become a general recipe layered on any strong global encoder.
  • The paper's cluster pseudo-labels are essentially a quantization of CLIP's region feature manifold; one could test a testable extension where the cluster centers are learned end-to-end instead of frozen, or re-clustered periodically, to see whether object semantics improve beyond the CLIP ceiling.
  • The random negative sampling rate ρ=0.1 and the use of only 10 sampled regions per image suggest a high degree of redundancy in both region data and negative classes; ablating region sample diversity (e.g., by sampling the largest or most salient regions) might reveal whether small, hard-detectable regions benefit more from the region supervision.
  • The unified loss treats OCR tokens as positive classes; an implicit corollary is that the same framework could be extended to other dense annotation types (e.g., depth or keypoint tokens) as positive classes, but the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces RICE, a region-aware cluster discrimination method for visual representation learning. It builds a billion-scale region dataset from SAM masks and PaddleOCR text, assigns object pseudo-labels by k-means on frozen CLIP features (Eq. 1), and introduces a Region Transformer layer with a mask-guided attention mechanism. The training loss combines a single-label object cluster discrimination loss and a multi-label OCR loss. Experiments evaluate RICE as a vision encoder in LLaVA-NeXT and LLaVA-OneVision frameworks, on referring segmentation, detection, tracking, and show hyperparameter ablations. The paper claims RICE consistently outperforms prior methods across segmentation, dense detection, and visual perception for MLLMs.

Significance. If the results hold, RICE provides a single vision encoder that improves region-level object semantics and OCR in one forward pass, with strong results on OCR-centric benchmarks and referring segmentation. The controlled LLaVA-NeXT setup with Qwen2.5-7B reduces some comparison bias, and the authors release the pretrained models, which supports reproducibility. The scale of pretraining (13B samples) is substantial. However, the object supervision is inherited from CLIP features, so the claimed object-semantic improvement is partly a distillation of CLIP geometry rather than an independent signal. The empirical gains still appear meaningful, especially at 336/378px, but the headline claim of 'consistently outperforms' is not supported by the paper's own Table 1 at 560px.

major comments (5)
  1. [Abstract, Sec. 5, Table 1] The claim that RICE 'consistently outperforms previous methods' is contradicted by Table 1. At 560px, RICE ViT-L-14-560px loses to Qwen2.5-ViT-ViT-H-14-560px on 9 of 15 benchmarks (e.g., TextVQA -4.74, OCRBench -55, MMStar -4.51) and loses to SigLIPv2-560px on 8 of 15 benchmarks, including the 'Other Avg' aggregate (-0.14). The text in Sec. 4.2 selectively highlights wins on InfoVQA and DocVQA while omitting the negative differences. Please rephrase the claim to specify the regimes where RICE is superior (e.g., OCR-centric tasks and lower resolutions) and report an aggregate or significance test that supports any global statement.
  2. [Sec. 3.3, Eqs. (3) and (4)] The loss formulations are incomplete and thus not reproducible. Eq. (3) as written is 'Lobject = log(1 + exp(-sim(...)) + log(1 + sum ...' with mismatched parentheses; the second log term has no closing parenthesis and the negative-class sum is not clearly inside the log. Eq. (4) has the same problem. Please give precise, well-parenthesized definitions, and also specify the sampling procedure for negative centers in the object loss (the reference to [2] is insufficient without the exact uniform-sampling subset defined in the text).
  3. [Sec. 4.5] The ablation study covers hyperparameters (N, K, rho, M) but does not isolate the main components of the method: the Region Transformer layer, the object cluster discrimination loss, and the OCR loss. Without removing each component in turn, the gains cannot be attributed to the proposed mechanisms. Add component ablations on at least one detection and one OCR benchmark, e.g., training with only the OCR loss, only the object loss, and with global attention instead of region attention.
  4. [Sec. 3.1, Eq. (1)] The object pseudo-labels are produced by k-means on frozen CLIP features, so the object supervision is a distillation of CLIP's feature geometry. The paper should state this explicitly and analyze its implication: the improvement over CLIP on object-centric tasks may come from region-level training and OCR labels rather than from genuinely better object semantics. Provide an experiment that separates these factors, such as training RICE with random or uniform region labels (or with the object branch removed) to measure the marginal contribution of the cluster-based object loss.
  5. [Tables 1-4] All results are reported from single runs with no variance estimates or statistical significance. Many differences in Table 1 are within a few tenths of a point (e.g., RICE-560 vs SigLIPv2-560 on POPE is -0.32). For a claim of 'consistently outperforms', please report standard deviations over at least 2-3 seeds for the principal comparisons (Table 1 and Table 3), or use paired tests where feasible.
minor comments (5)
  1. [Table 5a] Column headers in Table 5a are '1 5 10 20' while the row label says 'Testset'; these are values of the sampled-region count N, not test sets. Rename the column header to 'N' for clarity.
  2. [Eq. (1)] The text says features are extracted 'using the CLIP model' without specifying which variant (e.g., ViT-L/14, OpenCLIP, or Data Filtering Networks). Specify the exact CLIP checkpoint used for pseudo-label generation.
  3. [Sec. 4.1] The phrase 'processing a total of 13 billion samples during the initial pretraining stage' is ambiguous: it is unclear whether this is the number of unique samples, the number of samples seen per epoch, or the cumulative number over all epochs. Clarify the relation to dataset sizes.
  4. [Abstract and Sec. 5] The term 'previous methods' is used broadly but the comparison set includes Qwen2.5-ViT, which is described as a specialized backbone. Specify in the abstract or introduction whether the claim covers only methods available at the time of submission or also contemporaneous specialized models.
  5. [Eq. (3)-(4)] Define the similarity function sim(·) explicitly (e.g., cosine similarity after L2 normalization) and note the margin m and scale s from Sec. 4.1, which do not appear in the loss equations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RICE's object supervision distills CLIP cluster geometry, but the central claims are externally benchmarked and self-citations are methodological, not load-bearing.

full rationale

The paper's derivation chain is not circular. Object region labels are produced by k-means on frozen CLIP features (Eq. 1), so RICE's object supervision is a distillation of CLIP's feature geometry; however, this is not a predicted quantity that the paper claims to derive from first principles. The central claims are empirical: RICE is evaluated as a frozen encoder on external benchmarks (COCO/LVIS detection and segmentation, refCOCO referring segmentation, DocVQA/OCRBench/MMStar, etc.) against other pretrained encoders. These evaluations are outside the training labels and not entailed by the cluster-assignment construction. Self-citations to UNICOM [3] and MLCD [4] supply the k-means labeling and multi-label cluster discrimination losses, but they are methodological precedents, not uniqueness theorems, and the region-level architecture plus OCR branch are independently described. The abstract's 'consistently outperforms' is contradicted by Table 1 at 560px against Qwen2.5-ViT and SigLIPv2 on several benchmarks, but that is an overclaim/correctness risk rather than circularity. No load-bearing step reduces by construction to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The ledger shows that the object branch is built entirely on CLIP's own feature space, while OCR supervision comes from PaddleOCR text. The main hyperparameters K, rho, N, and M are chosen empirically and ablated only on a 10% data subset. No new physical or semantic entities are postulated beyond the learned cluster centers, which are internal pseudo-labels rather than independently evidenced constructs.

free parameters (5)
  • Number of cluster centers K = 1,000,000
    The number of semantic object clusters is chosen empirically; Table 5b shows COCO peaking at 1M and LVIS-Det at 2M.
  • Negative sampling rate rho = 0.1
    Set to 0.1 following UNICOM and MLCD, but Table 5c shows COCO-Det is best at 0.05 and other metrics at 0.1.
  • Sampled regions per image N = 10
    Table 5a shows 10 sampled boxes performs best on detection and segmentation, but no value is given in Sec. 3.2.
  • Positive OCR labels M = not explicitly stated
    Table 5d tests M in {3, 5, 10, 20}; the value used in the final model is not stated in the method section.
  • Margin m and scale s = m = 0.3, s = 64
    Adopted from margin-based classification and prior cluster discrimination work; these are chosen hyperparameters, not derived from first principles.
assumptions (5)
  • ad hoc to paper CLIP region features are semantically coherent enough that k-means over them yields valid object labels.
    Eq. (1) defines all object supervision through frozen CLIP features; no independent label source is used.
  • domain assumption SAM mask proposals correspond to meaningful object regions.
    SAM-generated masks are used as region proposals in Sec. 3.1 without evaluation of mask semantic quality.
  • domain assumption PaddleOCR transcriptions with confidence above 0.7 are sufficiently accurate as OCR training targets.
    Sec. 3.1 filters OCR entries by confidence but does not measure the noise rate or its effect on the learned representations.
  • ad hoc to paper Fixed cluster centers computed before training remain valid supervision throughout training.
    The paper does not re-cluster or update centers during training, so the target space is frozen at the CLIP initialization.
  • domain assumption Masked attention with -inf outside the region in Eq. (2) produces region representations that preserve semantic fidelity.
    The region visibility mask is a design choice; the paper provides only empirical, not analytical, support for its superiority.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Region-based Cluster Discrimination for Visual Representation Learning." pith.science (2026). https://pith.science/paper/WSYMD6LD

@misc{pith2026250720025,
  author       = {Pith},
  title        = {Pith review of: Region-based Cluster Discrimination for Visual Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSYMD6LD}},
  note         = {Machine review of arXiv:2507.20025}
}
read the original abstract

Learning visual representations is foundational for a broad spectrum of downstream tasks. Although recent vision-language contrastive models, such as CLIP and SigLIP, have achieved impressive zero-shot performance via large-scale vision-language alignment, their reliance on global representations constrains their effectiveness for dense prediction tasks, such as grounding, OCR, and segmentation. To address this gap, we introduce Region-Aware Cluster Discrimination (RICE), a novel method that enhances region-level visual and OCR capabilities. We first construct a billion-scale candidate region dataset and propose a Region Transformer layer to extract rich regional semantics. We further design a unified region cluster discrimination loss that jointly supports object and OCR learning within a single classification framework, enabling efficient and scalable distributed training on large-scale data. Extensive experiments show that RICE consistently outperforms previous methods on tasks, including segmentation, dense detection, and visual perception for Multimodal Large Language Models (MLLMs). The pre-trained models have been released at https://github.com/deepglint/MVT.

Figures

Figures reproduced from arXiv: 2507.20025 by the authors.

Figure 1
Figure 1. Visualization of object feature distributions from the COCO test dataset via t-SNE projection onto a spherical manifold. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our unified semantic region understanding framework. Our approach efficiently processes diverse semantic regions [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The region attention module processes batches of size [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Token distance distributions observed during the training [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Tracking PCA. Using 2048-resolution images as input to a ViT-B/16 model, we project token features onto RGB channels via [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval

    cs.CV 2025-09 conditional novelty 6.0 of 10

    GA-DMS with the WebPerson dataset sets new state-of-the-art Rank-1 accuracy on CUHK-PEDES, ICFG-PEDES, and RSTPReid.

Reference graph

Works this paper leans on

85 extracted references · 60 canonical work pages · cited by 1 Pith paper

  1. [2]

    Killing Two Birds with One Stone: Efficient and Robust Training of Face Recogni- tion CNNs by Partial FC

    Xiang An, Jiankang Deng, Jia Guo, Ziyong Feng, XuHan Zhu, Jing Yang, and Tongliang Liu. Killing Two Birds with One Stone: Efficient and Robust Training of Face Recogni- tion CNNs by Partial FC. In CVPR, 2022. 5

    2022

  2. [1]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a Visual Language Model for Few-Shot Learning. InNeurIPS,

  3. [3]

    Unicom: Universal and Compact Representation Learning for Image Retrieval

    Xiang An, Jiankang Deng, Kaicheng Yang, Jiawei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and Compact Representation Learning for Image Retrieval. In ICLR, 2023. 1, 2, 3, 4, 5

    2023

  4. [4]

    Multi-label Cluster Discrimination for Vi- sual Representation Learning

    Xiang An, Kaicheng Yang, Xiangzi Dai, Ziyong Feng, and Jiankang Deng. Multi-label Cluster Discrimination for Vi- sual Representation Learning. In ECCV, 2024. 1, 2, 5, 6

    2024

  5. [5]

    Self-Labelling via Simultaneous Clustering and Representation Learning

    Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-Labelling via Simultaneous Clustering and Representation Learning. In ICLR, 2020. 1

    2020

  6. [6]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. 2023. 3

    work page 2023

  7. [7]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966, 2023. 3

    arXiv 2023

  8. [8]

    COYO-700M: Image-Text Pair Dataset, 2022

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. COYO-700M: Image-Text Pair Dataset, 2022. 3, 5

    work page 2022

Show all 85 references
  1. [9]

    Cascade R-CNN: Delv- ing Into High Quality Object Detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delv- ing Into High Quality Object Detection. In CVPR, 2018. 6

    2018

  2. [10]

    Deep Clustering for Unsupervised Learn- ing of Visual Features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep Clustering for Unsupervised Learn- ing of Visual Features. In ECCV, 2018. 1

    2018

  3. [11]

    Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In NeurIPS, 2020. 1

    2020

  4. [12]

    ViTamin: Designing Scalable Vision Models in the Vision-language Era

    Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. ViTamin: Designing Scalable Vision Models in the Vision-language Era. In CVPR, 2024. 1

    2024

  5. [13]

    Are We on the Right Way for Evaluating Large Vision-Language Models? In NeurIPS,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are We on the Right Way for Evaluating Large Vision-Language Models? In NeurIPS,

  6. [14]

    Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv:2412.05271, 2024. 7

    2024 arXiv

  7. [15]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023. 3, 7

    2023

  8. [16]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019. 5

    2019

  9. [17]

    The Faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazare, Maria Lomeli, Lucas Hosseini, and Herve Jegou. The Faiss library. arXiv:2401.08281, 2025. 3

    2025 arXiv

  10. [18]

    PP-OCR: A Practical Ultra Lightweight OCR System

    Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. PP-OCR: A Practical Ultra Lightweight OCR System. arXiv:2009.09941, 2020. 2, 3

    2009 arXiv

  11. [19]

    Susskind, and Armand Joulin

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel ´Angel Bautista, Vaishaal Shankar, Alexander T To- shev, Joshua M. Susskind, and Armand Joulin. Scalable Pre-training of Large Autoregressive Image Models. PMLR,

  12. [20]

    Lasot: A high-quality benchmark for large-scale single ob- ject tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single ob- ject tracking. In CVPR, 2019. 7

    2019

  13. [21]

    Data Filtering Networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data Filtering Networks. In ICLR, 2024. 1, 6

    2024

  14. [22]

    Multimodal autoregres- sive pre-training of large vision encoders

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor G Turrisi da Costa, Louis B ´ethune, Zhe Gan, Alexander Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua Susskind, and Alaaeldin El-Nouby. Multimodal autoregres- ...

    2025

  15. [23]

    Rwkv-clip: A robust vision-language representation learner

    Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, and Jiankang Deng. Rwkv-clip: A robust vision-language representation learner. In EMNLP,

  16. [24]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask R-CNN. In ICCV, 2017. 6

    2017

  17. [25]

    GOT-10k: A Large High-Diversity Benchmark for Generic Object Track- ing in the Wild

    Lianghua Huang, Xin Zhao, and Kaiqi Huang. GOT-10k: A Large High-Diversity Benchmark for Generic Object Track- ing in the Wild. TPAMI, 2019. 7

    2019

  18. [26]

    Qwen2.5-Coder Technical Report

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayi- heng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024. 5

    2024 arXiv

  19. [27]

    Open- CLIP

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- CLIP. 2021. 1, 7

    2021

  20. [28]

    ReferItGame: Referring to Objects in Pho- tographs of Natural Scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to Objects in Pho- tographs of Natural Scenes. In EMNLP, 2014. 7

    2014

  21. [29]

    A Diagram Is Worth A Dozen Images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A Diagram Is Worth A Dozen Images. In ECCV, 2016. 6

    2016

  22. [30]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. In ICCV, 2023. 2, 3, 5

    2023

  23. [31]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. In CVPR, 2024. 3, 5, 7

    2024

  24. [32]

    LLaV A-OneVision: Easy Visual Task Trans- fer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. LLaV A-OneVision: Easy Visual Task Trans- fer. arXiv:2408.03326, 2024. 1, 3

    2024 arXiv

  25. [33]

    LLaV A-NeXT- Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun MA, and Chunyuan Li. LLaV A-NeXT- Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models. In ICLR, 2025. 1, 5

    2025

  26. [34]

    BLIP: Bootstrapping Language-Image Pre-training for Uni- fied Vision-Language Understanding and Generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Uni- fied Vision-Language Understanding and Generation. In ICML, 2022. 3

    2022

  27. [35]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In ICML, 2023. 3

    2023

  28. [36]

    Grounded Language-Image Pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded Language-Image Pre-training. In CVPR, 2022. 2

    2022

  29. [37]

    Evaluating Object Hallucination in Large Vision-Language Models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating Object Hallucination in Large Vision-Language Models. In EMNLP, 2023. 6

    2023

  30. [38]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. In CVPR, 2024. 1, 3

    2024

  31. [39]

    LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Im- proved reasoning, OCR, and world knowledge, 2024. 5

    2024

  32. [40]

    MMBench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281, 2023. 6

    2023 arXiv

  33. [41]

    OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models. Science China Informa- tion Sciences, 2024. 6

    2024

  34. [42]

    A ConvNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In CVPR, 2022. 7

    2022

  35. [43]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2018. 5

    2018

  36. [44]

    ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In ACL Findings, 2022. 6

    2022

  37. [45]

    DocVQA: A Dataset for VQA on Document Images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A Dataset for VQA on Document Images. In WACV, 2021. 6

    2021

  38. [46]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, 2022. 6

    2022

  39. [47]

    Trackingnet: A large-scale dataset and benchmark for object tracking in the wild

    Matthias Muller, Adel Bibi, Silvio Giancola, Salman Al- subaihi, and Bernard Ghanem. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV, 2018. 7

    2018

  40. [48]

    DINOv2: Learning Robust Visual Features without Supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michae...

    2024

  41. [49]

    Filtering, Dis- tillation, and Hard Negatives for Vision-Language Pre- Training

    Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. Filtering, Dis- tillation, and Hard Negatives for Vision-Language Pre- Training. In CVPR, 2023. 1

    2023

  42. [50]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML,

  43. [51]

    Pre- Det: Large-Scale Weakly Supervised Pre-Training for De- tection

    Vignesh Ramanathan, Rui Wang, and Dhruv Mahajan. Pre- Det: Large-Scale Weakly Supervised Pre-Training for De- tection. In ICCV, 2021. 2

    2021

  44. [52]

    GLaMM: Pixel Grounding Large Multimodal Model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. GLaMM: Pixel Grounding Large Multimodal Model. In CVPR, 2024. 3, 7

    2024

  45. [53]

    You Only Look Once: Unified, Real-Time Object Detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You Only Look Once: Unified, Real-Time Object Detection. In CVPR, 2016. 2

    2016

  46. [54]

    Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NeurIPS, 2015. 2

    2015

  47. [55]

    PixelLM: Pixel Reasoning with Large Multimodal Model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. PixelLM: Pixel Reasoning with Large Multimodal Model. In CVPR, 2024. 3, 7

    2024

  48. [56]

    LAION- 400M: Open Dataset of CLIP-Filtered 400 Million Image- Text Pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION- 400M: Open Dataset of CLIP-Filtered 400 Million Image- Text Pairs. arXiv:2111.02114, 2021. 5

    2021 arXiv

  49. [57]

    LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. arXiv:2210.08402, 2022. 3

    arXiv

  50. [58]

    LiveXiv– A Multi-Modal Live Benchmark Based on Arxiv Papers Content

    Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, M Jehanzeb Mirza, Leshem Chosen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, et al. LiveXiv– A Multi-Modal Live Benchmark Based on Arxiv Papers Content. arXiv:2410.10783, 2024. 6

    arXiv

  51. [59]

    Towards VQA Models That Can Read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read. In CVPR, 2019. 6

    2019

  52. [60]

    EV A-CLIP: Improved Training Techniques for CLIP at Scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: Improved Training Techniques for CLIP at Scale. arXiv:2303.15389, 2023. 1

    2023 arXiv

  53. [61]

    InternLM2 Technical Report

    Intern Team. InternLM2 Technical Report. arXiv:2403.17297, 2024. 7

    2024 arXiv

  54. [62]

    LLaMA2: Open Foundation and Fine-Tuned Chat Models

    LLaMA Team. LLaMA2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, 2023. 3, 7

    2023 arXiv

  55. [63]

    Qwen2 Technical Report

    Qwen Team. Qwen2 Technical Report. ArXiv:2407.10671,

    arXiv

  56. [64]

    Qwen2.5-VL Technical Report

    Qwen Team. Qwen2.5-VL Technical Report. arXiv:2502.13923, 2025. 6

    2025 arXiv

  57. [65]

    Cambrian-1: A Fully Open, Vision-Centric Ex- ploration of Multimodal LLMs

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A Fully Open, Vision-Centric Ex- ploration of Multimodal LLMs. In NeurIPS, 2024. 3

    2024

  58. [66]

    SigLIP 2: Multilingual Vision- Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H ´enaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual Vision- Languag...

    2025

  59. [67]

    Towards more flexible and accurate object tracking with natural language: Algo- rithms and benchmark

    Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algo- rithms and benchmark. In CVPR, 2021. 7

    2021

  60. [68]

    LaSagnA: Language-based Segmentation Assistant for Complex Queries

    Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. LaSagnA: Language-based Segmentation Assistant for Complex Queries. arXiv:2404.08506, 2024. 3

    2024 arXiv

  61. [69]

    VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Ping Luo, Yu Qiao, and Jifeng Dai. VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks. InNeurIPS,

  62. [70]

    CLIM: Contrastive Language-Image Mosaic for Region Representation.arXiv:2312.11376, 2023

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy. CLIM: Contrastive Language-Image Mosaic for Region Representation.arXiv:2312.11376, 2023. 1, 2

    2023 arXiv

  63. [71]

    Detectron2, 2019

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2, 2019. 6

    2019

  64. [72]

    Grok-1.5 Vision Preview, 2024

    XAI.ORG. Grok-1.5 Vision Preview, 2024. 6

    2024

  65. [73]

    GSV A: Generalized Segmentation via Multimodal Large Language Models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSV A: Generalized Segmentation via Multimodal Large Language Models. In CVPR, 2023. 3, 7

    2023

  66. [74]

    Alip: Adaptive language-image pre-training with synthetic cap- tion

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic cap- tion. In ICCV, 2023. 2

    2023

  67. [75]

    Clip-cid: Efficient clip distillation via cluster-instance discrimination

    Kaicheng Yang, Tiancheng Gu, Xiang An, Haiqiang Jiang, Xiangzi Dai, Ziyong Feng, Weidong Cai, and Jiankang Deng. Clip-cid: Efficient clip distillation via cluster-instance discrimination. In AAAI, 2025. 2

    2025

  68. [76]

    Joint feature learning and relation modeling for tracking: A one-stream framework

    Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV, 2022. 7

    2022

  69. [77]

    A Survey on Multimodal Large Language Models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A Survey on Multimodal Large Language Models. arXiv:2306.13549, 2023. 6

    2023 arXiv

  70. [78]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling Context in Referring Expres- sions. In ECCV, 2016. 7

    2016

  71. [79]

    Sa2V A: Marrying SAM2 with LLaV A for Dense Grounded Understanding of Images and Videos

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2V A: Marrying SAM2 with LLaV A for Dense Grounded Understanding of Images and Videos. arXiv:2501.04001, 2025. 3, 7

    2025 arXiv

  72. [80]

    Sigmoid Loss for Language Image Pre- Training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Training. In ICCV, 2023. 1, 2, 6

    2023

  73. [81]

    LLaV A-Grounding: Grounded Visual Chat with Large Multimodal Models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, and Jainwei Yang. LLaV A-Grounding: Grounded Visual Chat with Large Multimodal Models. In ECCV, 2024. 3, 7

    2024

  74. [82]

    OMG-LLaV A: Bridging Image-level, Object-level, Pixel- level Reasoning and Understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. OMG-LLaV A: Bridging Image-level, Object-level, Pixel- level Reasoning and Understanding. In NeurIPS, 2024. 7

    2024

  75. [83]

    RegionCLIP: Region-based Language-Image Pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. RegionCLIP: Region-based Language-Image Pretraining. In CVPR, 2022. 1, 2

    2022

  76. [84]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024. 3

    2024

  77. [85]

    UNIT: Unifying Image and Text Recognition in One Vision Encoder

    Yi Zhu, Zhou Yanpeng, Chunwei Wang, Yang Cao, Jianhua Han, Lu Hou, and Hang Xu. UNIT: Unifying Image and Text Recognition in One Vision Encoder. In NeurIPS, 2024. 2, 6

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.