Pith. sign in

REVIEW 4 major objections 5 minor 62 references

ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Re-captioning images before detection makes instance-grounded text-to-image generation markedly better.

desk verdict The pre-detection re-captioning idea is genuinely new and the dataset is a real artifact, but the headline grounding gain is partly a measurement loop you should not take at face value. read the letter →

arxiv 2508.01008 v1 pith:OPJ5UZAQ submitted 2025-08-01 cs.CV

Cihang Peng , Qiming Hou , Zhong Ren , Kun Zhou This is my paper
classification cs.CV
keywords text-to-imagegenerationinstancegroundingopen-vocabularydetectiondatasetconstructionVLM-LLMre-captioningboundingboxannotationsvision-languagemodellargelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ROVI, a dataset of 1M curated web images annotated for instance-grounded text-to-image generation, and argues that the key to better annotations is to re-caption images before detection rather than after. A VLM writes a comprehensive visual description of each image, an LLM converts it into a flat list of candidate object categories, and four open-vocabulary detectors then find boxes for those categories. Because the categories come from the image rather than from terse web captions, ROVI captures secondary objects and attributes that previous pipelines miss, and it grows the vocabulary to over 1.4 million distinct categories. The paper demonstrates the payoff by training the same GLIGEN generator on ROVI and showing that it outperforms state-of-the-art alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality, including in a user study.

What carries the argument

The load-bearing mechanism is pre-detection VLM-LLM re-captioning: before any open-vocabulary detector runs, InternVL1.5 writes a dense description of every recognizable object and its composition, and Llama3 reduces that description, together with the original web caption, into a flat list of candidate categories in two passes, one preserving compound phrases for attribute binding and one decomposing them into basic nouns for recall. Four detectors — Grounding-DINO, YOLO-World, OWLv2, and OV-DINO — each receive this same list; their boxes are merged with IoU-based NMS, resampled to reduce overlap and redundancy, and then verified by a second VLM (Qwen2VL) that answers a yes/no question about whether each crop matches its label. This pipeline is what gives the detectors a comprehensive open-vocabulary inventory that includes secondary elements such as bedding, pillows, and blanket, plus attributes like lamp with black shade, so the training signal contains the objects and details that web captions omit.

What would settle it

Re-run both the dataset filtering and the grounding evaluation with an independent verifier, for example human annotators labeling a random sample of the 24,000 inspected boxes or a second, differently trained VLM with a calibrated threshold, and compare acceptance rates between ROVI-generated images and the comparison methods; if the 0.880 vs 0.750-style gaps shrink or reverse, the reported grounding advantage is an artifact of the shared Qwen2VL oracle.

Watch

Extended reading notes

Core claim

The central discovery is that the pre-detection stage, not the detector or the generator, is where instance-grounded training data loses most of its information. The paper claims that generating a comprehensive VLM description of each image and then having an LLM summarize it into a flat candidate list produces a global prompt that is inherently linked to the instance annotations, capturing elements humans typically overlook. On 5K sampled images this procedure yields on average 12.45 categories and 24.01 boxes per image, compared with 0.72 categories and 0.86 boxes for web captions passed directly to a phrase-grounding detector, and it discovers 25,374 distinct categories versus 3,001. Trained on the full 1M-image ROVI dataset, a GLIGEN model reaches a per-box grounding pass rate of 0.880 on ROVI validation and 0.872 on Open Images validation, and it beats official GLIGEN, MIGC, and InstanceDiffusion on FID, aesthetic score, CLIP similarity, and pairwise user preference in instance alignment, prompt alignment, and aesthetics.

Load-bearing premise

The load-bearing premise is that the Qwen2VL yes/no check reliably tells whether a cropped image matches its label, in both the filtering step that builds ROVI and the evaluation step that reports grounding accuracy; if that model is systematically permissive or biased toward ROVI-style labels, the training labels and the headline pass rates rise together.

Editorial extensions

If this is right

  • ROVI grows the category vocabulary from a few thousand to 1,443,360 distinct categories across 1,012K images, with per-image averages of 12.51 categories and 24.21 boxes, enabling instance grounding at open-vocabulary scale.
  • On a 5K-image sample, the pre-detection re-captioning pipeline discovers 25,374 categories and 24.01 boxes per image, compared with 3,001 categories and 0.86 boxes for web captions fed to a phrase-grounding detector, and it beats RAM/RAM++ tagging pipelines on both counts.
  • A GLIGEN trained on ROVI reaches per-box grounding pass rates of 0.880 on ROVI validation and 0.872 on Open Images validation, higher than official GLIGEN (0.750/0.812), MIGC (0.821/0.833), and InstanceDiffusion (0.854/0.871).
  • The same ROVI-trained model improves image-level quality and prompt fidelity: FID 15.7/16.7, aesthetic score 5.83/5.40, and CLIP similarity 0.286/0.245, beating all comparison methods on both validation sets.
  • In a paired user study, ROVI-trained GLIGEN is preferred over official GLIGEN, MIGC, and InstanceDiffusion in instance alignment (82.0–94.8%), prompt alignment (82.3–90.5%), and aesthetic quality (83.3–98.3%).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The re-captioning strategy should transfer beyond GLIGEN: because the gain lives in the training annotations rather than the generator architecture, other grounded generation models, including higher-resolution diffusion and transformer-based generators, could expect similar improvements from training on ROVI or re-running the pipeline on their own images.
  • The grounding metric and the training filter share the same Qwen2VL verifier, so the reported 0.88 pass rate and the comparison gaps are partly self-measured; an independent verifier or human labels on a random sample would reveal how much of the advantage is real versus an artifact of the shared oracle.
  • The flat category list produced by the LLM could be reused as detector input for tasks beyond generation, such as retrieval, captioning, or segmentation pseudo-labeling, since it is model-agnostic and only requires re-running the open-source VLM/LLM/OVD stack on new images.
  • The paper's focus on pre-detection suggests a cheap design principle for future grounded-generation datasets: spend annotation effort on what the detector sees, not on post-hoc per-box captioning, because web captions and per-box VLM captions both omit the global compositional context that the global prompt needs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ROVI, a 1M-image synthetic dataset for instance-grounded text-to-image generation, built by re-captioning web images with a VLM (InternVL1.5), summarizing the descriptions into flat open-vocabulary category lists with an LLM (Llama3), detecting instances with four open-vocabulary detectors, and cross-checking each box-label pair with a second VLM (Qwen2VL). The authors train a GLIGEN model on ROVI and report that it outperforms existing methods in instance grounding accuracy, prompt fidelity, and aesthetic quality on both the ROVI validation set and the Open Images validation set, supported by a user study. The pipeline and dataset are released with reproducible code and open-source models.

Significance. If the reported gains hold, ROVI would be a valuable public resource: it demonstrates a practical pre-detection re-captioning strategy, combines multiple open-vocabulary detectors, and releases a large-scale dataset with far more categories and richer per-instance labels than existing detection-oriented datasets. The paper is also commendable for shipping the dataset, code, and intermediate pipeline outputs, and for reporting compute costs. The user study provides independent human preference evidence for the generation-quality claims. However, the headline instance-grounding metric and the aesthetic metric are each tied to the dataset-construction filters, so the quantitative evidence for the central claim is currently weaker than the abstract suggests.

major comments (4)
  1. [Section 3.5 and Section 4.2, Table 5] The instance grounding pass rate ('Gen Inst.' in Table 5) is measured with the same Qwen2VL yes/no cross-check that was used to filter the training boxes in Section 3.5. The training filter removes instances whose labels Qwen2VL does not approve, and the evaluation then scores generated crops with the same oracle; this creates a measurement loop in which the reported 0.880/0.872 pass rates may largely reflect self-consistency between the label-checker and the generator rather than independent grounding quality. The paper does not validate Qwen2VL's judgments against human annotators on generated images; the manual inspection in Section 5 covers source images only and reports a 3.3% error rate without measuring human agreement with Qwen2VL. To support the claim of improved grounding accuracy, the authors should re-evaluate on a human-annotated sample of generated images, use a different VLM not involved in dataset construction, or report agreement and threshold-sensitivity analyses for the Qwen2VL checker.
  2. [Section 3.1 and Table 5] The aesthetic score (Aes.) in Table 5 is computed with the same LAION aesthetic scorer used to curate the training images in Section 3.1 (threshold 5.75). Because the ROVI training set is explicitly selected to score highly on this metric, the higher Aes. values for the ROVI-trained model may be inflated by distributional alignment with the scorer rather than by genuinely better image aesthetics. The FID and CLIP-Sim results are less affected, but the aesthetic-quality claim needs an independent evaluation, such as a human preference study with formal analysis or a different aesthetic model that was not part of the curation pipeline.
  3. [Section 4.1, Table 4] Table 4 reports the number of categories and boxes for each data-generation method 'only count[ing] box-caption pairs that passed our VLM cross-check in Sec. 3.5.' If the Qwen2VL checker is systematically more permissive for ROVI-style labels (e.g., because they are derived from VLM descriptions and LLM summarization similar to the checker's training distribution), this counting rule could inflate ROVI's apparent advantage over the comparison methods. The paper should report the raw detection counts before the cross-check, and ideally verify the cross-check on a per-method sampled subset with human labels, so the comparison does not depend on the same oracle used in both data construction and evaluation.
  4. [Section 4.2, Table 6 and Appendix F] The user study is the most independent evidence for the paper's claims, but the reporting is too thin to carry that weight. Table 6 gives only win-rate percentages; the paper does not state the number of participants, the number of judgments per pair, inter-rater agreement, or any statistical significance test. Appendix F describes the interface and the decision to recruit trusted expert volunteers, but it does not report how many users completed the study or how ties were handled. The authors should provide these details, and ideally a stratified analysis of instance-alignment judgments, to substantiate the claim that human observers prefer the ROVI-trained model for grounding.
minor comments (5)
  1. [Title and Figure 5] The title contains a typo ('Open-V ocabulary') and Figure 5 contains 'keeling' instead of 'kneeling'; these should be corrected.
  2. [Section 3.2] The statement that 'we dedicate approximately half of our input tokens to explicitly instruct the VLM' is vague; please specify the exact prompting protocol, token budget, or provide the prompt template in the appendix.
  3. [Section 3.4 and Appendix C] The resampling details are deferred to the supplementary materials and 'the codes coming'; for a self-contained manuscript, at least the penalty terms and the five-layer sampling procedure should be summarized in the main text or appendix.
  4. [Section 2.1] The related-work paragraph on text-to-image generation is very brief and does not mention several recent layout/grounding methods; consider expanding or adding a sentence that directly positions ROVI relative to methods like Ranni, BoxDiff, or IFAdapter, which are cited later but not compared experimentally.
  5. [Table 5] The table caption calls the comparison 'apples-to-apples,' but the compared models are trained on different datasets and different pipelines; the caption should clarify that only the evaluation inputs are identical, while training data differ, to avoid overstating the comparison's fairness.

Circularity Check

1 steps flagged · score 5.0 of 10

Grounded-metric measurement loop: the Qwen2VL yes/no checker that filters ROVI's training boxes is reused as the 'Gen Inst.' evaluation metric, so the headline grounding pass rate partly measures agreement with the dataset's own label oracle.

  1. self definitional [Sec. 3.5 (VLM Cross-Checking) and Sec. 4.2 / Table 5]
    "We subsequently prompt a second VLM, Qwen2VL [49], to check whether the image content matches its expected caption. ... We only keep an instance when the total probability of all capitalization variants of “yes” is significantly more than that of “no”. ... At the instance level, we focus on instance grounding accuracy, evaluated using the VLM-based cross-checking method in Sec. 3.5. Specifically, we cropped each expected instance from its specified bounding box on a generated image and used Qwen2VL [49] to compute the likelihood of the cropped window matching its assigned label."

    The 'Gen Inst.' pass rate is Qwen2VL yes/no agreement, and ROVI's training boxes were kept only when that same Qwen2VL yes/no test passed. So the evaluation measures the generator's agreement with the exact oracle used to construct the labels. Any permissive bias in Qwen2VL, or any bias toward VLM/LLM-style label phrasing, inflates both the training label set and the reported 0.880/0.872 scores together. The paper's manual inspection of 24k boxes examines source images, not generated images, and does not calibrate Qwen2VL's threshold on outputs; hence the grounding metric is not an independent measurement. The human user study (Table 6) provides independent support, but the automated headline grounding numbers are partially circular.

full rationale

The paper's core contribution—pre-detection VLM-LLM re-captioning to feed flat category lists to open-vocabulary detectors—is a logically independent pipeline design, and the dataset and code are openly released. The central claim is not defined into existence, and the human user study (Table 6) gives independent evidence that ROVI-trained GLIGEN improves instance alignment, prompt alignment, and aesthetics. However, the headline quantitative grounding metric 'Gen Inst.' in Table 5 is the same Qwen2VL yes/no cross-check used in Sec. 3.5 to filter which boxes and labels enter ROVI. The evaluation therefore measures agreement with the very oracle that pre-selected the training labels; if Qwen2VL is permissive or biased toward VLM/LLM-generated phrasings, the filter and the scorer inflate together. The paper's manual inspection of 24k boxes (Sec. 5) covers only source images and does not calibrate Qwen2VL on generated images, so it does not break the loop. A secondary concern is that the aesthetic metric is the same LAION aesthetic predictor used for source-image curation, though the Open Images validation comparison and external human judgments mitigate this. Overall, one of the three headline metrics reduces to a measurement loop, while the other evidence keeps the finding from being fully circular; hence score 5.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about the reliability of the VLM/LLM/OVD stack and on hand-chosen curation thresholds. No new physical or conceptual entities are postulated; the categories are extracted from existing images and web captions. The thresholds shape the dataset statistics and are manually set, not fitted to an external benchmark.

free parameters (5)
  • aesthetic score cutoff = 5.75
    Chosen in Section 3.1 to filter COYO and LAION images; only 2.1% of images pass, so it strongly shapes the dataset and the aesthetic evaluation that uses the same scorer.
  • minimum resolution = 1024 x 1024
    Curation threshold in Section 3.1; drives the reported average resolution of 2102 x 1488.
  • pHash deduplication Hamming distance = 10
    Deduplication threshold in Section 3.1; removes 31% of redundant images.
  • NMS IoU threshold = 0.4
    Used in Section 3.4 to merge boxes from four OVDs; affects final instance counts and category statistics.
  • resampling layer count = 5
    Appendix C sets a sampling target of five lower-overlap layers, removing approximately 70% of boxes and influencing annotation density.
assumptions (4)
  • domain assumption InternVL1.5 descriptions are accurate and comprehensive enough to serve as the category source for detection.
    The re-captioning stage in Section 3.2 assumes this; Appendix D documents known VLM errors such as mirrored left/right judgments and hallucinated details.
  • domain assumption Llama3 summarization maps descriptive text to valid open-vocabulary categories.
    Section 3.3 relies on the LLM to extract categories; Appendix D shows it produces questionable categories such as "blue dressed woman" and misattributes identities.
  • domain assumption The Qwen2VL yes/no question is a valid oracle for box-label alignment.
    Section 3.5 uses it to accept or reject every box, and Section 4.2 uses the same method to measure grounding accuracy. If Qwen2VL is biased, both the dataset and the evaluation inherit the bias.
  • domain assumption The four OVDs, combined with IO U-NMS and resampling, preserve the true object instances in each image.
    Section 3.4 assumes detection recall after merging is sufficient; the paper's own manual check finds 3.3% of 24k boxes erroneous (Section 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation." pith.science (2026). https://pith.science/paper/OPJ5UZAQ

@misc{pith2026250801008,
  author       = {Pith},
  title        = {Pith review of: ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPJ5UZAQ}},
  note         = {Machine review of arXiv:2508.01008}
}
read the original abstract

We present ROVI, a high-quality synthetic dataset for instance-grounded text-to-image generation, created by labeling 1M curated web images. Our key innovation is a strategy called re-captioning, focusing on the pre-detection stage, where a VLM (Vision-Language Model) generates comprehensive visual descriptions that are then processed by an LLM (Large Language Model) to extract a flat list of potential categories for OVDs (Open-Vocabulary Detectors) to detect. This approach yields a global prompt inherently linked to instance annotations while capturing secondary visual elements humans typically overlook. Evaluations show that ROVI exceeds existing detection datasets in image quality and resolution while containing two orders of magnitude more categories with an open-vocabulary nature. For demonstrative purposes, a text-to-image model GLIGEN trained on ROVI significantly outperforms state-of-the-art alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality. Our dataset and reproducible pipeline are available at https://github.com/CihangPeng/ROVI.

Figures

Figures reproduced from arXiv: 2508.01008 by the authors.

Figure 1
Figure 1. Effect of our pre-detection strategy. Image (a) shows a web image with original caption where OVDs only detect [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline overview: stages of our systematic instance annotation framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Challenges identified in VLM/LLM inference trials. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Effect of RAM++ tagging plus per-box VLM captioning. RAM++ tags: [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparing model performance on compositional prompts with instance grounding. Besides, note the attribute binding issues [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Effect of pre-detection VLM-LLM re-captioning, our key contribution. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The above presents a comparison of the generated results. Note that the GLIGEN model trained on ROVI ( [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: In the results mentioned above, the prompt in the third row requests the generation of a [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: The superior visual quality and coherence demonstrated in our generated images can be attributed to the rigorous curation [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Landscapes constitute a significant proportion of our high-aesthetic dataset. Traditional detection approaches with restricted [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Human activities present significant challenges for traditional bounding box detection approaches. Users consistently require [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Indoor scenes involve numerous objects and complex visual hierarchies that challenge traditional detection methods. The [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Food categories present particular complexity in object-centric detection and constitute a significant proportion of social media [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Our dataset encompasses stylized images from diverse internet sources, where detailed VLM descriptions demonstrate partic [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Effect of OVDs after the pre-detection VLM-LLM re-captioning. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Heatmaps on categories of concern. Here, we discuss how to integrate the detection results of OVDs from a fundamental perspective, specifically by examining the overlap relationships between bounding boxes. According to the frequency relationships revealed in [PITH_F…
Figure 17
Figure 17. Figure 17: Box candidates in layers of resampling in relatively low overlap. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Effect of a limited number of resampled and cross-checked boxes. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Effect of dense open-vocabulary detection based on VLM-LLM re-captioning. [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: LLM summarized questionable open-vocabulary categories. [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Discontinuous bounding box segments of edge and gray plate [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]
Figure 22
Figure 22. Figure 22: In the image above, the left side of the bounding box input represents a building, while the right side depicts a person, [PITH_FULL_IMAGE:figures/full_fig_p025_22.png]
Figure 23
Figure 23. Figure 23: The category small bonsai tree is a low-frequency open-vocabulary category within our dataset, which may hinder the potentials of corresponding generative capabilities. In this example, the official GLIGEN’s generation results for “bonsai” may be closer to expecta￾tio…
Figure 24
Figure 24. Figure 24: In this set of examples, the angel in the upper group and the butter spread in the lower group were not generated correctly. This is likely attributable to the relatively low-frequency visual associations present in the training dataset. For instance, all models in th…
Figure 25
Figure 25. Figure 25: This set of examples illustrates several visual elements that extend beyond the spatial representation capabilities of bounding [PITH_FULL_IMAGE:figures/full_fig_p027_25.png]
Figure 26
Figure 26. Figure 26: A screenshot of our user study interface. [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 28 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Multidiffusion: Fusing diffusion paths for controlled image generation, 2023

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation, 2023. 1

    work page 2023

  3. [3]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https : / / github . com / kakaobrain/coyo-dataset, 2022. 4

    work page 2022

  4. [4]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. 2

    arXiv 2024

  5. [5]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 2

    arXiv

  6. [6]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353, 2024. 3

    work page 2024

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 2, 4, 9

    arXiv 2024

  8. [8]

    Yolo-world: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 2, 3, 4, 5, 11

    work page 2024

Show all 62 references
  1. [9]

    Laion pop: 600,000 high-resolution images with detailed descriptions

    Peter Bevan Christoph Schuhmann. Laion pop: 600,000 high-resolution images with detailed descriptions. https: //laion.ai/blog/laion-pop, 2023. 4

    2023

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  3. [11]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  4. [12]

    Ranni: Taming text-to-image diffu- sion for accurate instruction following

    Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffu- sion for accurate instruction following. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4744–4753, 2024. 2

    2024

  5. [13]

    Im- ageinwords: Unlocking hyper-detailed image descriptions

    Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bun- ner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Im- ageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024. 3

    2024 arXiv

  6. [14]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 2

    2024 arXiv

  7. [15]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 2

    2020

  8. [16]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 2

    2025 arXiv

  9. [17]

    LVIS: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 1, 2, 3

    2019

  10. [18]

    Retrieval-augmented open-vocabulary object detec- tion

    Jooyeon Kim, Eulrang Cho, Sehyung Kim, and Hyunwoo J Kim. Retrieval-augmented open-vocabulary object detec- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 17427–17436,

  11. [19]

    Dense text-to-image generation with attention modulation

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV, 2023. 1

    2023

  12. [20]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  13. [21]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 2, 8

    2024

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  15. [23]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  16. [24]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. arXiv preprint arXiv:2406.11838, 2024. 2

    2024 arXiv

  17. [25]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023. 1, 2, 3, 5, 6, 7, 9

    2023

  18. [26]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  19. [27]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014 , pages 740–755, Cham,

    2014

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  21. [29]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 2, 3, 5, 11

    2023 arXiv

  22. [30]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neu- ral Information Processing Systems , 36, 2024. 2, 3, 4, 5, 11

    2024

  23. [31]

    Docci: De- scriptions of connected and contrasting images, 2024

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: De- scriptions of connected and contrasting images, 2024. URL https://arxiv. org/abs/2404.19753. 3

    2024 arXiv

  24. [32]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn. openai.com/papers/GPTV_System_Card.pdf ,

  25. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  26. [34]

    Grounded text-to-image synthesis with attention refocusing

    Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7932–7942, 2024. 3

    2024

  27. [35]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2

    2023 arXiv

  28. [36]

    Stanza: A python natural language processing toolkit for many human languages.arXiv preprint arXiv:2003.07082, 2020

    Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. Stanza: A python natural language processing toolkit for many human languages.arXiv preprint arXiv:2003.07082, 2020. 4

    2003 arXiv

  29. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  30. [38]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 2

    2021

  31. [39]

    Stable diffusion v1.4 checkpoint

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. Stable diffusion v1.4 checkpoint. https : / / github . com / CompVis / stable-diffusion, 2022. 6

    2022

  32. [40]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 9

    2022

  33. [41]

    Laion-aesthetics

    Christoph Schuhmann. Laion-aesthetics. https : / / laion.ai/blog/laion-aesthetics, 2022. 4, 6

    2022

  34. [42]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  35. [43]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 8430–8439, 2019. 2, 3

    2019

  36. [44]

    From pixels to prose: A large dataset of dense image cap- tions

    Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels to prose: A large dataset of dense image cap- tions. arXiv preprint arXiv:2406.10328, 2024. 3

    2024 arXiv

  37. [45]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 2

    2024 arXiv

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  39. [47]

    Ov-dino: Unified open-vocabulary de- tection with language-aware selective fusion

    Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang, Xiangyuan Lan, et al. Ov-dino: Unified open-vocabulary de- tection with language-aware selective fusion. arXiv preprint arXiv:2407.07844, 2024. 2, 3, 4, 5, 11

    2024 arXiv

  40. [48]

    V3det: Vast vocabulary visual detection dataset

    Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 19844–19854, 2023. 2, 3

    2023

  41. [49]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 5, 7

    2024 arXiv

  42. [50]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 2

    2023 arXiv

  43. [51]

    Instancediffusion: Instance-level control for image generation, 2024

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation, 2024. 1, 2, 3, 4, 5, 6, 7, 9

    2024

  44. [52]

    Ifadapter: Instance feature con- trol for grounded text-to-image generation

    Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, and Xinchao Wang. Ifadapter: Instance feature con- trol for grounded text-to-image generation. arXiv preprint arXiv:2409.08240, 2024. 1, 2, 4, 5

    2024 arXiv

  45. [53]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452–7461, 2023. 3

    2023

  46. [54]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 2

    2024 arXiv

  47. [55]

    Detclipv3: To- wards versatile generative open-vocabulary object detection

    Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipv3: To- wards versatile generative open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27391–27401, 2024. 2, 3

    2024

  48. [56]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022. 2

    2022 arXiv

  49. [57]

    Glipv2: Unifying localiza- tion and vision-language understanding

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq- Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localiza- tion and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067–36080, 2022. 3

    2022

  50. [58]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 17...

    2024

  51. [59]

    Improving pseudo labels for open-vocabulary object detection

    Shiyu Zhao, Samuel Schulter, Long Zhao, Zhixing Zhang, Yumin Suh, Manmohan Chandraker, Dimitris N Metaxas, et al. Improving pseudo labels for open-vocabulary object detection. arXiv preprint arXiv:2308.06412, 2023. 2, 3

    2023 arXiv

  52. [60]

    Migc++: Advanced multi-instance generation controller for image synthesis, 2024

    Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Migc++: Advanced multi-instance generation controller for image synthesis, 2024. 1, 2, 3, 5

    2024

  53. [61]

    Migc: Multi-instance generation controller for text-to-image synthesis, 2024

    Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Migc: Multi-instance generation controller for text-to-image synthesis, 2024. 1, 2, 3, 5, 7, 9

    2024

  54. [2014]

    Springer International Publishing. 1, 2, 3

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.