Pith. sign in

REVIEW 4 major objections 6 minor 55 references

Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that pixel-level visual entity linking becomes tractable by reversing the annotation direction: start from known entity labels, segment their masks, and train on the resulting 5.2M-annotation dataset to reach 25.2%…

desk verdict Real new task and a substantial dataset, but the 94.8% quality headline is an entity-balanced estimate that needs to be re-checked example-weighted before it is trusted. read the letter →

arxiv 2412.13614 v1 pith:TFZP6MP5 submitted 2024-12-18 cs.CV cs.AIcs.CLcs.IRcs.MM

Zhengfei Xu , Sijia Zhao , Yanchao Hao , Xiaolong Liu , Lili Li , Yuyang Yin , Bo Li , Xi Chen
show 1 more author
Xin Xin
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.IRcs.MM
keywords visualentitylinkingpixel-levelreverseannotationMaskOVEN-Wikidatasetsemantictokenizationregion-interactedattentionopen-vocabularysegmentationmultimodallargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Pixel-Level Visual Entity Linking (PL-VEL): given an image and a pixel mask that points at one object, the model must name that object's entity among millions of Wikipedia entries. The central claim is that building training data for this task is feasible through reverse annotation: instead of asking a model to identify a segmented region, start from entity labels already paired with images in the OVEN-Wiki benchmark and use open-vocabulary segmentation models to draw each entity's mask. Knowledge-augmented text references (Wikidata hypernyms plus GPT-3.5-derived referring expressions), a model ensemble, and heuristic filtering raise annotation accuracy to 94.8% on a 2,000-sample manual check, yielding the 5.2M-annotation MaskOVEN-Wiki dataset. A model fine-tuned on this dataset—with a new visual semantic tokenization that turns segment-everything regions into tokens—improves linking accuracy from 1.3% (zero-shot) to 25.2%, about 5 points of which come from the tokenization. The upshot is that pixel masks offer a more practical reference than elaborate text queries for fine-grained visual understanding.

What carries the argument

The load-bearing mechanism is the reverse annotation framework. Given a triple (entity, image, text query) from OVEN-Wiki, it constructs an enriched text reference: intensional expansion adds Wikidata super-categories (instance-of and subclass-of) to the entity label, and extensional expansion uses GPT-3.5 to extract the query's spatial or relational referring expressions. This reference is fed to two open-vocabulary segmentation models—the pipeline Grounded-SAM (Grounding-DINO plus SAM) and the end-to-end SEEM—whose mask proposals are then corrected by heuristic rules: drop non-visual entity types, treat location and building entities as whole-image masks, use IOU agreement to catch pipeline error propagation, and apply morphological erosion/dilation followed by connected-component analysis to fix foreground-background confusion in dense scenes. On the modeling side, the second load-bearing piece is visual semantic tokenization: FastSAM runs 'segment everything' on the image; Osprey's mask-aware visual extractor encodes each resulting region into a semantic and a positional embedding; the region tokens are sorted by descending area (simulating coarse-to-fine human attention) and concatenated with the patch features of the vision encoder before entering the LLM. The model generates a 4-token ALD code for the target entity, following the generative entity-recognition framework of GER-ALD, with Vicuna as the language backbone fine-tuned by LoRA.

What would settle it

Take a fresh random sample of about 1,000 annotations from the full MaskOVEN-Wiki, stratified by entity category, and have independent annotators check mask-region alignment under a pre-registered rubric; or, alternatively, have humans draw masks for a few hundred of the same (image, entity) pairs and compare how a model fine-tuned on MaskOVEN-Wiki performs with those human masks versus the pipeline masks. If the category-stratified accuracy falls well below 94.8%, or accuracy drops sharply with human-drawn masks, the central claim that reverse annotation produces a high-quality large-scale data resource would be weakened.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is an asymmetry: direct region-to-entity annotation is hard (even GPT-4V reaches only about 25.5% accuracy on the OVEN-Wiki labeling task), while the reverse direction—entity-to-region segmentation—is tractable (about 81% with a plain Grounded-SAM pipeline, raised to 94.8% with knowledge augmentation, ensembling, and filtering). Exploiting that asymmetry yields a dataset of 5,245,421 masks aligned with 20,077 entities across 5,214,965 images, drawn from the existing OVEN-Wiki entities, queries, and Wikipedia-image splits. The paper further claims that training on this data works: Osprey-7B goes from 1.3% zero-shot accuracy to 20.0% after fine-tuning, and adding their visual semantic tokenization—which replaces fixed-size patches with semantic region tokens produced by FastSAM—reaches 25.2% overall accuracy on the test set. The authors present PL-VEL as a task definition, a dataset, and a baseline, arguing that pixel-mask references supplement text-based visual entity linking and enable finer-grained visual understanding in multimodal LLMs.

Load-bearing premise

The load-bearing premise is that the 94.8% annotation-success rate measured on 2,000 hand-checked samples represents the quality of the full 5.2M-annotation dataset; the heuristic filters that discard location, building, and sports entities are what keep the sample looking clean, and the validity of every downstream accuracy number depends on that small sample being representative.

Editorial extensions

If this is right

  • PL-VEL becomes a new benchmark: the MaskOVEN-Wiki dataset supplies train/validation/test splits in entity, query, and wiki variants plus a human-evaluated set, all with pixel-mask references.
  • Pixel masks are a practical stand-in for text queries: mask-based linking reaches accuracy comparable to strong text-based models (25.2% vs. 25.5% for GPT-4V) and outperforms box references (1.6%), suggesting masks carry richer spatial information.
  • Fine-tuning on MaskOVEN-Wiki is what turns a zero-shot model into a functional linker—1.3% to 20.0%—so the dataset, not the model architecture alone, drives the gain.
  • The reverse-annotation recipe generalizes beyond this dataset: knowledge augmentation plus model ensembling raised annotation accuracy from 81% to 94.8%, a pipeline that can be applied to other region-labeling efforts.
  • Retrieval augmentation is an open direction: a retrieval-free text VEL model performs within 0.5 points of the proposed model, implying that adding retrieval-based entity search to PL-VEL could close much of the remaining gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The entity-to-region asymmetry suggests a general recipe for open-vocabulary annotation: when the label space is millions of classes, generating the visual evidence conditioned on a known label is far easier than classifying an unlabeled region; the same trick could apply to building data for other fine-grained recognition tasks.
  • The reported 94.8% annotation success rests on a 2,000-sample check that is balanced by entity rather than by image, and the filtering rules deliberately drop location, building, and sports entities; an independent, category-stratified re-estimate would be needed to confirm the figure holds across the full 5.2M dataset.
  • Because the final accuracy is only about 25%, the masks themselves may carry a systematic bias (easy entities, filtered hard categories), meaning gains measured on MaskOVEN-Wiki might not transfer to arbitrary user-provided masks; a small evaluation with human-drawn masks on the same images would test transfer.
  • Visual semantic tokenization is a portable idea: segment-everything regions could replace fixed patches in any region-grounded multimodal LLM task—region captioning, referring expression comprehension, VQA—so its 5-point gain here is a plausible lower bound for other fine-grained tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Pixel-Level Visual Entity Linking (PL-VEL), where an object in an image is referred to by a pixel mask rather than a text query, and the goal is to link that visual mention to an entity in a large knowledge base. To support this task, the authors construct MaskOVEN-Wiki by reversing the direction of the existing OVEN-Wiki dataset: given an entity and a text query, they prompt open-vocabulary segmentation models (Grounded-SAM and SEEM) with knowledge-augmented textual references, then apply heuristic filtering and correction rules. They report a 94.8% annotation-success rate based on a 2,000-item manual evaluation and a final dataset with 5,245,421 annotations. They also propose a visual semantic tokenization scheme that adds FastSAM region features (sorted by area) to the patch features of an Osprey-based MLLM, and they report that fine-tuning on MaskOVEN-Wiki raises test accuracy from 1.3% (zero-shot) to 25.2%, with semantic tokenization contributing about 5 points.

Significance. If the quality estimate is reliable, this is a useful contribution: the paper proposes a new task, releases a large-scale pixel-level entity linking dataset, and provides a baseline that shows mask-based references can approach text-based VEL performance. The reverse annotation rationale is coherent and avoids the prohibitive 6M-entity search space of direct recognition. The paper is also transparent about the main filtering categories and makes code and data available. The central weakness is that the headline 94.8% annotation-quality figure is not yet established for the actual example-weighted dataset, and the downstream model comparisons inherit this uncertainty. The paper does not appear to contain circular derivations; training uses external labels from OVEN-Wiki and a separate evaluation split.

major comments (4)
  1. [Annotation Quality / Table 6] The 94.8% annotation-success estimate is entity-balanced, not example-weighted. Table 6 shows that in the manual evaluation every entity contributes exactly one sample (Case Num equals Entity Num in each split), while Table 2 lists 4,464,748 Entity-Split training examples over 7,943 seen entities, i.e., roughly 560 examples per entity on average. If annotation success correlates with entity frequency or with the entity types that dominate the example count, the entity-balanced rate need not equal the per-annotation quality of the released 5.2M masks. Please report accuracy separately for each split, by entity-frequency stratum, and with confidence intervals; ideally, re-run the manual evaluation with example-weighted sampling.
  2. [Annotation Quality / Definition of success] The paper does not disclose the rubric used in the manual evaluation. It is not stated what counts as a successful annotation: e.g., whether the mask must reach an IoU threshold with the target object, whether partial or occluded objects are accepted, or how cases where the entity is absent from the image are coded. No inter-annotator agreement measure is provided, and the single 'overall after filtering' number in Table 1 has no confidence interval. Without this operationalization, the headline quality claim cannot be independently checked.
  3. [Data Filtering / Figure 5(b)] The heuristic filtering rules induce a selection bias that is acknowledged qualitatively but not quantified. The Data Filtering section states that location, building, and sports entities have the highest proportion of unannotated entities, and the Appendix reports that the rules exclude 124,896 annotations in the Entity Split, 7,920 in the Query Split, and 176 in the Human Set. This changes the category composition of the final dataset relative to OVEN-Wiki. Moreover, the manual evaluation distribution in Figure 4 differs substantially from the final dataset distribution in Figure 5(a) for location (18.8% vs. 6.2%) and building (12.9% vs. 4.9%), so the evaluation set is not representative of the published data. Please report the category distribution before and after filtering and give per-category annotation accuracy, especially for the categories most affected by the filters.
  4. [Experiments / Table 3 and Table 4] The main text says visual semantic tokenization improves overall performance by 3.4% on the validation set and 5.2% on the test set, but the cited tables show a validation improvement of 3.7 points without pre-training (+FT+Seg vs. +FT) and 5.0 points with pre-training (+PT+FT+Seg vs. +PT+FT), and a test improvement of 5.2 points. The reported 3.4% does not match any number in the tables. This discrepancy should be corrected and the exact comparison used for the headline '5-point improvement' claim should be stated explicitly.
minor comments (6)
  1. [Main Results] The sentence 'Osprey-7B-Seg improves the overall performance by 3.4% on the validation set' is inconsistent with the numbers in Table 3 and Table 4; please reconcile the figure or the tables.
  2. [Figure 5(b)] Figure 5(b) contains a file path and timestamp ('2024/6/11 12:14 Awesome-pyecharts file:///Users/...') and appears to be a screenshot; it should be redrawn as a clean vector figure with labeled axes.
  3. [Figure 3 caption] The Figure 3 caption says the illustration image was generated by AI and cites (Chang et al. 2024), but that reference is a paper on GPU kernel fusion (FLUX), not an image-generation model; the citation is incorrect.
  4. [Table 1] The symbols in Table 1, such as '#' in '# entity label' and '# text query', are not defined, and the row 'overall after filtering' reports a single aggregate without explaining how the aggregate is computed across the settings and splits.
  5. [Table 2] Table 2 is difficult to read: the 'Human Set' label appears in the header but the human-set statistics are not clearly separated from the other columns; please align the column structure or add a note describing which splits are included.
  6. [Equation (3)] Equation (3) is referenced in the text but the displayed equation for the region feature set appears without a visible equation number; please check all equation numbering.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims rest on manual evaluation and held-out test splits, with no derivation reducing to its own inputs or to a self-citation chain.

full rationale

The paper's central claims are (i) that the reverse annotation pipeline achieves 94.8% accuracy, (ii) that models trained on the resulting MaskOVEN-Wiki dataset improve linking accuracy, and (iii) that visual semantic tokenization adds about 5 points. None of these reduces to the paper's own construction by definition. The 94.8% figure comes from manual inspection of 2,000 randomly sampled annotations (Table 6), which is an external ground-truth check rather than a fitted parameter; the heuristic filtering rules are described as applied before the evaluation, and the paper does not state that the evaluation set was used to tune them. The model gains (1.3% to 25.2%, and +5 points for segmentation tokens) are measured on held-out validation and test splits of the constructed dataset, with the zero-shot comparison providing an independent baseline. The method builds on cited external components (OVEN-Wiki, Grounded-SAM, SEEM, Osprey, FastSAM) without invoking a load-bearing self-citation chain. The only residual concern is statistical rather than circular: the entity-balanced 2,000-sample manual evaluation may not exactly represent the example-weighted 5.2M-annotation dataset, but that is a measurement-estimation issue, not a derivation that is equivalent to its inputs. Therefore no circular step meeting the evidentiary bar for this analysis is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims rest on external tools and source datasets, not on new theoretical constructs. The main assumptions are the reliability of OVEN-Wiki labels, the fidelity of segmentation models, the correctness of knowledge augmentation, and the representativeness of the manual evaluation.

assumptions (5)
  • domain assumption OVEN-Wiki provides correct entity-image-query alignments.
    The entire MaskOVEN-Wiki is built on OVEN-Wiki; if the source labels are wrong, the masks inherit errors (see Dataset Construction).
  • domain assumption Segmentation models (Grounded-SAM, SEEM) produce masks that align with the intended entities.
    The reverse annotation pipeline relies on these models to generate accurate masks for the entity prompts; their errors are expected to be corrected by filtering.
  • domain assumption Wikidata hypernyms and GPT-3.5 extracted referring expressions improve segmentation query quality without introducing new errors.
    The knowledge augmentation step assumes that Wikidata instance-of/subclass-of relations and GPT-3.5 extractions are semantically correct for the target entities.
  • domain assumption The ALD code representation (from GER-ALD) is effective for autoregressive entity identification.
    The model relies on the ALD encoding as a compact target for the LLM; this is borrowed from prior work without re-validation in this setting.
  • domain assumption The 2,000-sample manual evaluation is representative of the full dataset's annotation quality.
    The 94.8% accuracy claim generalizes from this sample to the entire 5.2M-annotation dataset, but no confidence interval or stratified sampling details are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking." pith.science (2026). https://pith.science/paper/TFZP6MP5

@misc{pith2026241213614,
  author       = {Pith},
  title        = {Pith review of: Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TFZP6MP5}},
  note         = {Machine review of arXiv:2412.13614}
}
read the original abstract

Visual Entity Linking (VEL) is a crucial task for achieving fine-grained visual understanding, matching objects within images (visual mentions) to entities in a knowledge base. Previous VEL tasks rely on textual inputs, but writing queries for complex scenes can be challenging. Visual inputs like clicks or bounding boxes offer a more convenient alternative. Therefore, we propose a new task, Pixel-Level Visual Entity Linking (PL-VEL), which uses pixel masks from visual inputs to refer to objects, supplementing reference methods for VEL. To facilitate research on this task, we have constructed the MaskOVEN-Wiki dataset through an entirely automatic reverse region-entity annotation framework. This dataset contains over 5 million annotations aligning pixel-level regions with entity-level labels, which will advance visual understanding towards fine-grained. Moreover, as pixel masks correspond to semantic regions in an image, we enhance previous patch-interacted attention with region-interacted attention by a visual semantic tokenization approach. Manual evaluation results indicate that the reverse annotation framework achieved a 94.8% annotation success rate. Experimental results show that models trained on this dataset improved accuracy by 18 points compared to zero-shot models. Additionally, the semantic tokenization method achieved a 5-point accuracy improvement over the trained baseline.

Figures

Figures reproduced from arXiv: 2412.13614 by the authors.

Figure 1
Figure 1. Overview of comparing text and pixel-based Vi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the annotation framework. (a) Comparison of direct and reverse annotation shows that direct annota [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The procedure of building MaskOVEN-Wiki. The illustration image is generated by AI (Chang et al. 2024). As illustrated in fig. 3, we have developed a knowledge￾enhanced methodology for segmentation annotation. This workflow consists of three steps: text reference construction, mask annotation, and data filtering. For automated pixel￾mask annotation, we utilize Grounded-SAM (Ren et al. 2024) and SEEM (Zou et al. 2023… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Entity category distribution in the evaluation set. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Model overview including 1) pre-built ALD codes [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Examples from MaskOVEN-Wiki entity split. Entity Image Pixel Mask Text Query Monkey King What is the name of the cartoon character on the left? Flat roof What type of roof does the smallest building have? Basset Hound What is the colloquial name for the breed of dog in…
Figure 8
Figure 8. Figure 8: Examples from MaskOVEN-Wiki query split. Experiment Details Annotation Setup We utilized a cluster of 30 nodes for the annotation of large￾scale data. Each node was configured with 7 CPU cores, 30 GB of memory, and an NVIDIA Tesla P40-24G GPU. For the MaskOVEN-Wiki dat…
Figure 9
Figure 9. Figure 9: Examples that triggered the filtering rules during the data annotation process. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Detailed statistics of unique entities grouped by source dataset on entity split (top red), query split (top blue), and [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 30 canonical work pages

  1. [1]

    Bossard, L.; Guillaumin, M.; and Van Gool, L. 2014. Food-101 -- Mining Discriminative Components with Random Forests. In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., Computer Vision -- ECCV 2014, 446--461. Cham: Springer International Publishing. ISBN 978-3-319-10599-4

    work page 2014

  2. [2]

    Caron, M.; Iscen, A.; Fathi, A.; and Schmid, C. 2024 a . A Generative Approach for Wikipedia-Scale Visual Entity Recognition . arxiv:2403.02041

    work page Pith review arXiv 2024

  3. [3]

    Caron, M.; Iscen, A.; Fathi, A.; and Schmid, C. 2024 b . A Generative Approach for Wikipedia-Scale Visual Entity Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17313--17322

    work page 2024

  4. [4]

    Chang, L.-W.; Bao, W.; Hou, Q.; Jiang, C.; Zheng, N.; Zhong, Y.; Zhang, X.; Song, Z.; Yao, C.; Jiang, Z.; Lin, H.; Jin, X.; and Liu, X. 2024. FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion. arXiv:2406.06858

    arXiv 2024

  5. [5]

    Chen, K.; and Wu, X. 2024. VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 27218--27227

    work page 2024

  6. [6]

    Chen, K.; Zhang, Z.; Zeng, W.; Zhang, R.; Zhu, F.; and Zhao, R. 2023. Shikra: Unleashing Multimodal LLM 's Referential Dialogue Magic . arxiv:2306.15195

    arXiv 2023

  7. [7]

    E.; Stoica, I.; and Xing, E

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\

    2023

  8. [8]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

Show all 55 references
  1. [9]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2020. An Image Is Worth 16x16 Words : Transformers for Image Recognition at Scale . In International Confe...

    2020

  2. [10]

    Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6325--6334

    2017

  3. [11]

    C.; Yu, Y.; Luo, P.; and Liu, S

    Guo, Q.; De Mello, S.; Yin, H.; Byeon, W.; Cheung, K. C.; Yu, Y.; Luo, P.; and Liu, S. 2024. Regiongpt: Towards region understanding vision language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13796--13806

    2024

  4. [12]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2016

  5. [13]

    J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  6. [14]

    Hu, H.; Luan, Y.; Chen, Y.; Khandelwal, U.; Joshi, M.; Lee, K.; Toutanova, K.; and Chang, M.-W. 2023. Open-Domain Visual Entity Recognition : Towards Recognizing Millions of Wikipedia Entities . In Proceedings of the IEEE / CVF International Conference on Computer Vision , 120...

    2023

  7. [15]

    Huang, X.; Zhang, Y.; Ma, J.; Tian, W.; Feng, R.; Zhang, Y.; Li, Y.; Guo, Y.; and Zhang, L. 2024. Tag2Text: Guiding Vision-Language Model via Image Tagging. In The Twelfth International Conference on Learning Representations

    2024

  8. [16]

    C.; Lo, W.-Y.; Doll \'a r, P.; and Girshick, R

    Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Doll \'a r, P.; and Girshick, R. 2023. Segment Anything . arxiv:2304.02643

    2023 arXiv

  9. [17]

    Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013. 3D Object Representations for Fine-Grained Categorization. In 2013 IEEE International Conference on Computer Vision Workshops, 554--561

    2013

  10. [18]

    A.; Bernstein, M

    Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; Bernstein, M. S.; and Fei-Fei, L. 2017. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vision, 12...

    2017

  11. [19]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. BLIP : Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation . In Proceedings of the 39th International Conference on Machine Learning , 12888--12900. PMLR

    2022

  12. [20]

    J.; Bourdev, L

    Lin, T.; Maire, M.; Belongie, S. J.; Bourdev, L. D.; Girshick, R. B.; Hays, J.; Perona, P.; Ramanan, D.; Doll ' a r, P.; and Zitnick, C. L. 2014. Microsoft COCO: Common Objects in Context. CoRR, abs/1405.0312

    2014 arXiv

  13. [21]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 a . Visual Instruction Tuning . In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 34892--34916. Curran Associates, Inc

    2023

  14. [22]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 b . Visual Instruction Tuning . arxiv:2304.08485

    2023 arXiv

  15. [24]

    Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Li, C.; Yang, J.; Su, H.; Zhu, J.; et al. 2023 d . Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499

    2023 arXiv

  16. [25]

    Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. 2022. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11976--11986

    2022

  17. [26]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations

    2019

  18. [27]

    Maji, S.; Rahtu, E.; Kannala, J.; Blaschko, M.; and Vedaldi, A. 2013. Fine-Grained Visual Classification of Aircraft. arXiv:1306.5151

    2013 arXiv

  19. [28]

    Marino, K.; Rastegari, M.; Farhadi, A.; and Mottaghi, R. 2019. OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3190--3199

    2019

  20. [29]

    Nilsback, M.-E.; and Zisserman, A. 2008. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 722--729

    2008

  21. [30]

    Peng, Z.; Wang, W.; Dong, L.; Hao, Y.; Huang, S.; Ma, S.; and Wei, F. 2023. Kosmos-2: Grounding Multimodal Large Language Models to the World . arxiv:2306.14824

    2023 arXiv

  22. [31]

    Piosenka, G. 2021. Sports100: 100 sports image classification. https://www.kaggle.com/datasets/gpiosenka/sports-classification. Accessed: 2022-09-26

    2021

  23. [32]

    A.; Xu, Y

    Qiu, J.; Madotto, A.; Lin, Z.; Crook, P. A.; Xu, Y. E.; Dong, X. L.; Faloutsos, C.; Li, L.; Damavandi, B.; and Moon, S. 2024. SnapNTell : Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM . arxiv:2403.04735

    2024 arXiv

  24. [33]

    S.; Shaker, A.; Khan, S.; Cholakkal, H.; Anwer, R

    Rasheed, H.; Maaz, M.; Mullappilly, S. S.; Shaker, A.; Khan, S.; Cholakkal, H.; Anwer, R. M.; Xing, E.; Yang, M.-H.; and Khan, F. S. 2023. GLaMM : Pixel Grounding Large Multimodal Model . arxiv:2311.03356

    2023 arXiv

  25. [34]

    M.; Xing, E.; Yang, M.-H.; and Khan, F

    Rasheed, H.; Maaz, M.; Shaji, S.; Shaker, A.; Khan, S.; Cholakkal, H.; Anwer, R. M.; Xing, E.; Yang, M.-H.; and Khan, F. S. 2024. GLaMM: Pixel Grounding Large Multimodal Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13009--13018

    2024

  26. [35]

    Rasley, J.; Rajbhandari, S.; Ruwase, O.; and He, Y. 2020. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, 3505–3...

    2020

  27. [36]

    Ren, T.; Liu, S.; Zeng, A.; Lin, J.; Li, K.; Cao, H.; Chen, J.; Huang, X.; Chen, Y.; Yan, F.; Zeng, Z.; Zhang, H.; Li, F.; Yang, J.; Li, H.; Jiang, Q.; and Zhang, L. 2024. Grounded SAM : Assembling Open - World Models for Diverse Visual Tasks . ArXiv:2401.14159 [cs]

    2024 arXiv

  28. [37]

    K.; Chowdhury, P

    Sain, A.; Bhunia, A. K.; Chowdhury, P. N.; Koley, S.; Xiang, T.; and Song, Y.-Z. 2023. CLIP for All Things Zero-Shot Sketch-Based Image Retrieval , Fine-Grained or Not . In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , 2765--2775

    2023

  29. [38]

    Saito, K.; Sohn, K.; Zhang, X.; Li, C.-L.; Lee, C.-Y.; Saenko, K.; and Pfister, T. 2023. Pic2Word : Mapping Pictures to Words for Zero-Shot Composed Image Retrieval . In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , 19305--19314

    2023

  30. [39]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards VQA Models That Can Read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8309--8318

    2019

  31. [40]

    Sun, W.; Fan, Y.; Guo, J.; Zhang, R.; and Cheng, X. 2022. Visual Named Entity Linking : A New Dataset and A Baseline . In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Findings of the Association for Computational Linguistics : EMNLP 2022 , 2403--2415. Abu Dhabi, United Ara...

    2022

  32. [41]

    Van Horn, G.; Mac Aodha, O.; Song, Y.; Cui, Y.; Sun, C.; Shepard, A.; Adam, H.; Perona, P.; and Belongie, S. 2018. The INaturalist Species Classification and Detection Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  33. [42]

    Wang, J.; Yang, Z.; Hu, X.; Li, L.; Lin, K.; Gan, Z.; Liu, Z.; Liu, C.; and Wang, L. 2022. GIT : A Generative Image-to-text Transformer for Vision and Language. Transactions on Machine Learning Research

    2022

  34. [43]

    Weyand, T.; Araujo, A.; Cao, B.; and Sim, J. 2020. Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval . In Proc. CVPR

    2020

  35. [44]

    J.; and Chen, E

    Wu, L.; Li, Z.; Zhao, H.; Wang, Z.; Liu, Q.; Huai, B.; Yuan, N. J.; and Chen, E. 2023. Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation . In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, 2618--262...

    2023

  36. [45]

    A.; Oliva, A.; and Torralba, A

    Xiao, J.; Hays, J.; Ehinger, K. A.; Oliva, A.; and Torralba, A. 2010. SUN database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3485--3492

    2010

  37. [46]

    Xiao, Z.; Gong, M.; Cascante-Bonilla , P.; Zhang, X.; Wu, J.; and Ordonez, V. 2024. Grounding Language Models for Visual Entity Recognition . arxiv:2402.18695

    2024 arXiv

  38. [47]

    Yuan, Y.; Li, W.; Liu, J.; Tang, D.; Luo, X.; Qin, C.; Zhang, L.; and Zhu, J. 2023. Osprey: Pixel Understanding with Visual Instruction Tuning . https://arxiv.org/abs/2312.10032v2

    2023 arXiv

  39. [48]

    Zhang, S.; Sun, P.; Chen, S.; Xiao, M.; Shao, W.; Zhang, W.; Liu, Y.; Chen, K.; and Luo, P. 2024 a . GPT4RoI : Instruction Tuning Large Language Model on Region-of-Interest . arxiv:2307.03601

    2024 arXiv

  40. [49]

    Zhang, Y.; Huang, X.; Ma, J.; Li, Z.; Luo, Z.; Xie, Y.; Qin, Y.; Luo, T.; Li, Y.; Liu, S.; et al. 2024 b . Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1724--1732

    2024

  41. [50]

    Zhang, Y.; Lin, C.; Cao, D.; and Lin, D. 2024 c . End- To-End Spatially-Constrained Multi-Perspective Fine-Grained Image Captioning . In ICASSP 2024 - 2024 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ) , 3360--3364

    2024

  42. [51]

    Zhao, X.; Ding, W.; An, Y.; Du, Y.; Yu, T.; Li, M.; Tang, M.; and Wang, J. 2023. Fast Segment Anything. arXiv:2306.12156

    2023 arXiv

  43. [52]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4 : Enhancing Vision-Language Understanding with Advanced Large Language Models . https://arxiv.org/abs/2304.10592v2

    2023 arXiv

  44. [53]

    Zhu, Y.; Groth, O.; Bernstein, M.; and Fei-Fei, L. 2016. Visual7W: Grounded Question Answering in Images. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4995--5004

    2016

  45. [54]

    Zou, X.; Yang, J.; Zhang, H.; Li, F.; Li, L.; Wang, J.; Wang, L.; Gao, J.; and Lee, Y. J. 2023. Segment Everything Everywhere All at Once . Advances in Neural Information Processing Systems, 36: 19769--19782

    2023

  46. [55]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  47. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.