{"id":"cd9dbd5e-1f5b-46ed-a5c2-5fd2ffb1b06f","arxiv_id":"2508.01008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ROVI's pre-detection VLM-LLM re-captioning yields richer open-vocabulary box labels on 1M curated images, and a GLIGEN model trained on ROVI improves instance grounding, prompt fidelity, and aesthetic quality in the paper's comparisons.","lead":"ROVI is a 1M-image dataset for text-to-image generation with detailed open-vocabulary object boxes, built by asking a vision model to describe each image and an LLM to turn the description into detection categories. A GLIGEN model trained on ROVI produces images that are grounded to boxes and prompts better than the baselines in the paper's comparisons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grounded-metric circularity: the same Qwen2VL yes/no verdict both filters ROVI labels and scores GLIGEN on them, so the reported 0.88 pass rate may largely measure label-checker agreement, not improved grounding.","rationale":"The reader's weakest_assumption is exactly the Qwen2VL oracle circularity, and my reading confirms it as the single most load-bearing concern. The paper's quantitative grounding claim rests entirely on Sec. 3.5's yes/no threshold applied at evaluation time, with no independent human calibration reported on generated images or on the relative ranking of systems. I considered whether the strongest claim could be rescued by the user study: Table 6 shows large win rates, but the study lacks reported n, participant independence, and blinding details (Sec. F), and its prompts were manually selected to be interpretable/meaningful, which reduces its power as an unbiased check of the same claim. The core dataset contribution (pre-detection re-captioning, category diversity, curation) remains plausible and useful, and the paper's own Sec. 5 limitation analysis acknowledges annotation errors (3.3%) and the inability of current GLIGEN to exploit high-resolution benefits. Therefore the verdict stays CONDITIONAL rather than REJECT: the central claim is genuinely unverified at the grounding metric, but the dataset and pipeline are released and the failure mode is empirically testable. No ad hominem concerns arise; the issue is a measurement assumption, not authorial integrity.","tokens_in":19795,"tokens_out":2649,"duration_ms":26664,"concrete_test":"Run a controlled human-annotation study on 200 sampled generated-image crops: for each method in Table 5 (SD, Offi. G., MIGC, InstDiff, Our G.), take the same 200 box-label inputs, crop the specified box from the generated image, and have 3 independent annotators (blinded to method) judge whether the crop matches the label. Compare the human pass rate to the Qwen2VL pass rate per method. If human pass rate for ROVI does not exceed alternatives (or differs from the reported 0.88 by >10 points), the grounding claim is inflated. As a second check, re-score the Table 5 generated images with a different, frozen VLM that was not used in dataset filtering; if the method ranking changes, the reported metric is checker-specific.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that GLIGEN trained on ROVI outperforms alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality (abstract, Table 5). Of these, the grounding comparison rests on a single oracle: Qwen2VL yes/no cross-checking (Sec. 3.5) is used to filter training boxes (keep only 'yes' significantly exceeding 'no') and is then reused in Sec. 4.2 to score generated images ('Gen Inst.' pass rate). This creates a measurement loop. If Qwen2VL is systematically permissive for ROVI-style labels, either because those labels are closer to its training distribution (VLM-derived descriptions parsed by an LLM) or because its yes/no calibration is biased toward 'yes', then (i) the dataset retains more boxes whose labels are only Qwen2VL-approved and (ii) the evaluation metric approves generated content against those same label phrasings. The two effects inflate together: the filtering step can remove exactly the labels that the scoring metric would fail, pre-selecting the evaluation distribution for the scoring metric. The paper reports no validation of Qwen2VL against human judgment on generated images (the Sec. 5 manual inspection of 24k boxes is for source images only and quantifies 3.3% errors, but does not measure human agreement with Qwen2VL nor calibrate the yes/no threshold). The claim '0.880/0.872 pass rate' is therefore better read as 'Qwen2VL agreement rate on generator outputs under a threshold tuned on Qwen2VL-approved training labels', not as independently established instance grounding accuracy. Because the user study (Table 6) was run by the authors without reported size or independence and grounded prompts were hand-picked 'to be interpretable', it does not remove the circularity risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ROVI, a 1M-image synthetic dataset for instance-grounded text-to-image generation, built by re-captioning web images with a VLM (InternVL1.5), summarizing the descriptions into flat open-vocabulary category lists with an LLM (Llama3), detecting instances with four open-vocabulary detectors, and cross-checking each box-label pair with a second VLM (Qwen2VL). The authors train a GLIGEN model on ROVI and report that it outperforms existing methods in instance grounding accuracy, prompt fidelity, and aesthetic quality on both the ROVI validation set and the Open Images validation set, supported by a user study. The pipeline and dataset are released with reproducible code and open-source models.","tokens_in":20130,"tokens_out":3486,"duration_ms":47839,"significance":"If the reported gains hold, ROVI would be a valuable public resource: it demonstrates a practical pre-detection re-captioning strategy, combines multiple open-vocabulary detectors, and releases a large-scale dataset with far more categories and richer per-instance labels than existing detection-oriented datasets. The paper is also commendable for shipping the dataset, code, and intermediate pipeline outputs, and for reporting compute costs. The user study provides independent human preference evidence for the generation-quality claims. However, the headline instance-grounding metric and the aesthetic metric are each tied to the dataset-construction filters, so the quantitative evidence for the central claim is currently weaker than the abstract suggests.","major_comments":[{"comment":"The instance grounding pass rate ('Gen Inst.' in Table 5) is measured with the same Qwen2VL yes/no cross-check that was used to filter the training boxes in Section 3.5. The training filter removes instances whose labels Qwen2VL does not approve, and the evaluation then scores generated crops with the same oracle; this creates a measurement loop in which the reported 0.880/0.872 pass rates may largely reflect self-consistency between the label-checker and the generator rather than independent grounding quality. The paper does not validate Qwen2VL's judgments against human annotators on generated images; the manual inspection in Section 5 covers source images only and reports a 3.3% error rate without measuring human agreement with Qwen2VL. To support the claim of improved grounding accuracy, the authors should re-evaluate on a human-annotated sample of generated images, use a different VLM not involved in dataset construction, or report agreement and threshold-sensitivity analyses for the Qwen2VL checker.","section":"Section 3.5 and Section 4.2, Table 5"},{"comment":"The aesthetic score (Aes.) in Table 5 is computed with the same LAION aesthetic scorer used to curate the training images in Section 3.1 (threshold 5.75). Because the ROVI training set is explicitly selected to score highly on this metric, the higher Aes. values for the ROVI-trained model may be inflated by distributional alignment with the scorer rather than by genuinely better image aesthetics. The FID and CLIP-Sim results are less affected, but the aesthetic-quality claim needs an independent evaluation, such as a human preference study with formal analysis or a different aesthetic model that was not part of the curation pipeline.","section":"Section 3.1 and Table 5"},{"comment":"Table 4 reports the number of categories and boxes for each data-generation method 'only count[ing] box-caption pairs that passed our VLM cross-check in Sec. 3.5.' If the Qwen2VL checker is systematically more permissive for ROVI-style labels (e.g., because they are derived from VLM descriptions and LLM summarization similar to the checker's training distribution), this counting rule could inflate ROVI's apparent advantage over the comparison methods. The paper should report the raw detection counts before the cross-check, and ideally verify the cross-check on a per-method sampled subset with human labels, so the comparison does not depend on the same oracle used in both data construction and evaluation.","section":"Section 4.1, Table 4"},{"comment":"The user study is the most independent evidence for the paper's claims, but the reporting is too thin to carry that weight. Table 6 gives only win-rate percentages; the paper does not state the number of participants, the number of judgments per pair, inter-rater agreement, or any statistical significance test. Appendix F describes the interface and the decision to recruit trusted expert volunteers, but it does not report how many users completed the study or how ties were handled. The authors should provide these details, and ideally a stratified analysis of instance-alignment judgments, to substantiate the claim that human observers prefer the ROVI-trained model for grounding.","section":"Section 4.2, Table 6 and Appendix F"}],"minor_comments":[{"comment":"The title contains a typo ('Open-V ocabulary') and Figure 5 contains 'keeling' instead of 'kneeling'; these should be corrected.","section":"Title and Figure 5"},{"comment":"The statement that 'we dedicate approximately half of our input tokens to explicitly instruct the VLM' is vague; please specify the exact prompting protocol, token budget, or provide the prompt template in the appendix.","section":"Section 3.2"},{"comment":"The resampling details are deferred to the supplementary materials and 'the codes coming'; for a self-contained manuscript, at least the penalty terms and the five-layer sampling procedure should be summarized in the main text or appendix.","section":"Section 3.4 and Appendix C"},{"comment":"The related-work paragraph on text-to-image generation is very brief and does not mention several recent layout/grounding methods; consider expanding or adding a sentence that directly positions ROVI relative to methods like Ranni, BoxDiff, or IFAdapter, which are cited later but not compared experimentally.","section":"Section 2.1"},{"comment":"The table caption calls the comparison 'apples-to-apples,' but the compared models are trained on different datasets and different pipelines; the caption should clarify that only the evaluation inputs are identical, while training data differ, to avoid overstating the comparison's fairness.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a useful dataset and a plausible pipeline, but the core quantitative claim of improved instance grounding currently rests on a metric that is coupled to the data-filtering oracle. I would like the revision to include an independent grounding evaluation (human or a different VLM) and fuller reporting of the user study; if those are supplied, the paper could be publishable. There is also a scope question: the paper primarily argues for a dataset contribution, but the experimental section is framed as a method-demonstration; the authors should make the primary contribution clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead ROVI. The core claim that this is the first to re-caption before detection is legitimate. The pipeline makes sense: InternVL writes a detailed description, Llama3 flattens it into a category list, four OVDs detect, Qwen2VL filters. That pre-detection step is what lets you use non-phrase-grounding detectors, and it captures secondary objects that web captions miss. The 1M image dataset with 1.44M distinct categories is a real contribution, and the release of the pipeline with open-source models means others can reproduce it. Credit where due: the paper is transparent about compute costs, the failure modes in the appendix are honest, and the manual inspection of 24k boxes gives some independent signal on label quality.\n\nThe soft spots matter. The grounding metric is circular: Qwen2VL yes/no is used to clean training boxes in Section 3.5 and then reused to score generated images in Section 4.2. So the 0.880 pass rate is at least partly a measure of how well the generator produces crops that the same checker likes, trained on labels the checker already approved. That does not kill the paper, but it means the \"grounding accuracy\" claim in the abstract is overstated. The aesthetic metric has an analogous problem, since the curation filter and the evaluation scorer are the same LAION model, though on the Open Images validation set the FID and CLIP gains are harder to explain away.\n\nThe other gaps: no controlled experiment where GLIGEN is trained on baseline labels (web caption + OVD or RAM) using the same code, so the improvement is not cleanly attributed to the dataset rather than to retraining. The user study is reported as win rates without number of participants or independence; the appendix says they recruited colleagues they trust, which is reasonable but not a rigorous evaluation.\n\nAll that said, the central idea is sound and the dataset is likely to be useful. The fix is not rejection but revision: validate the Qwen2VL check against human judgments on generated images, report the user study properly, and add the missing controlled training run. I'd send this to review, and I'd read the revised version carefully.","headline":"The pre-detection re-captioning idea is genuinely new and the dataset is a real artifact, but the headline grounding gain is partly a measurement loop you should not take at face value.","tokens_in":20745,"tokens_out":2591,"would_cite":true,"duration_ms":29285,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Re-captioning images before detection makes instance-grounded text-to-image generation markedly better.","keywords":["text-to-image generation","instance grounding","open-vocabulary detection","dataset construction","VLM-LLM re-captioning","bounding box annotations","vision-language model","large language model"],"falsifier":"Re-run both the dataset filtering and the grounding evaluation with an independent verifier, for example human annotators labeling a random sample of the 24,000 inspected boxes or a second, differently trained VLM with a calibrated threshold, and compare acceptance rates between ROVI-generated images and the comparison methods; if the 0.880 vs 0.750-style gaps shrink or reverse, the reported grounding advantage is an artifact of the shared Qwen2VL oracle.","tokens_in":19589,"feed_emoji":"🖼️","tokens_out":9278,"duration_ms":88805,"temperature":0.7,"pith_summary":"This paper introduces ROVI, a dataset of 1M curated web images annotated for instance-grounded text-to-image generation, and argues that the key to better annotations is to re-caption images before detection rather than after. A VLM writes a comprehensive visual description of each image, an LLM converts it into a flat list of candidate object categories, and four open-vocabulary detectors then find boxes for those categories. Because the categories come from the image rather than from terse web captions, ROVI captures secondary objects and attributes that previous pipelines miss, and it grows the vocabulary to over 1.4 million distinct categories. The paper demonstrates the payoff by training the same GLIGEN generator on ROVI and showing that it outperforms state-of-the-art alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality, including in a user study.","feed_headline":"New re-captioning recipe lifts instance-grounded image generation","feed_subtitle":"A GLIGEN trained on ROVI beats prior pipelines in grounding, fidelity, and aesthetics.","key_machinery":"The load-bearing mechanism is pre-detection VLM-LLM re-captioning: before any open-vocabulary detector runs, InternVL1.5 writes a dense description of every recognizable object and its composition, and Llama3 reduces that description, together with the original web caption, into a flat list of candidate categories in two passes, one preserving compound phrases for attribute binding and one decomposing them into basic nouns for recall. Four detectors — Grounding-DINO, YOLO-World, OWLv2, and OV-DINO — each receive this same list; their boxes are merged with IoU-based NMS, resampled to reduce overlap and redundancy, and then verified by a second VLM (Qwen2VL) that answers a yes/no question about whether each crop matches its label. This pipeline is what gives the detectors a comprehensive open-vocabulary inventory that includes secondary elements such as bedding, pillows, and blanket, plus attributes like lamp with black shade, so the training signal contains the objects and details that web captions omit.","core_discovery":"The central discovery is that the pre-detection stage, not the detector or the generator, is where instance-grounded training data loses most of its information. The paper claims that generating a comprehensive VLM description of each image and then having an LLM summarize it into a flat candidate list produces a global prompt that is inherently linked to the instance annotations, capturing elements humans typically overlook. On 5K sampled images this procedure yields on average 12.45 categories and 24.01 boxes per image, compared with 0.72 categories and 0.86 boxes for web captions passed directly to a phrase-grounding detector, and it discovers 25,374 distinct categories versus 3,001. Trained on the full 1M-image ROVI dataset, a GLIGEN model reaches a per-box grounding pass rate of 0.880 on ROVI validation and 0.872 on Open Images validation, and it beats official GLIGEN, MIGC, and InstanceDiffusion on FID, aesthetic score, CLIP similarity, and pairwise user preference in instance alignment, prompt alignment, and aesthetics.","pith_inferences":["The re-captioning strategy should transfer beyond GLIGEN: because the gain lives in the training annotations rather than the generator architecture, other grounded generation models, including higher-resolution diffusion and transformer-based generators, could expect similar improvements from training on ROVI or re-running the pipeline on their own images.","The grounding metric and the training filter share the same Qwen2VL verifier, so the reported 0.88 pass rate and the comparison gaps are partly self-measured; an independent verifier or human labels on a random sample would reveal how much of the advantage is real versus an artifact of the shared oracle.","The flat category list produced by the LLM could be reused as detector input for tasks beyond generation, such as retrieval, captioning, or segmentation pseudo-labeling, since it is model-agnostic and only requires re-running the open-source VLM/LLM/OVD stack on new images.","The paper's focus on pre-detection suggests a cheap design principle for future grounded-generation datasets: spend annotation effort on what the detector sees, not on post-hoc per-box captioning, because web captions and per-box VLM captions both omit the global compositional context that the global prompt needs."],"forward_implications":["ROVI grows the category vocabulary from a few thousand to 1,443,360 distinct categories across 1,012K images, with per-image averages of 12.51 categories and 24.21 boxes, enabling instance grounding at open-vocabulary scale.","On a 5K-image sample, the pre-detection re-captioning pipeline discovers 25,374 categories and 24.01 boxes per image, compared with 3,001 categories and 0.86 boxes for web captions fed to a phrase-grounding detector, and it beats RAM/RAM++ tagging pipelines on both counts.","A GLIGEN trained on ROVI reaches per-box grounding pass rates of 0.880 on ROVI validation and 0.872 on Open Images validation, higher than official GLIGEN (0.750/0.812), MIGC (0.821/0.833), and InstanceDiffusion (0.854/0.871).","The same ROVI-trained model improves image-level quality and prompt fidelity: FID 15.7/16.7, aesthetic score 5.83/5.40, and CLIP similarity 0.286/0.245, beating all comparison methods on both validation sets.","In a paired user study, ROVI-trained GLIGEN is preferred over official GLIGEN, MIGC, and InstanceDiffusion in instance alignment (82.0–94.8%), prompt alignment (82.3–90.5%), and aesthetic quality (83.3–98.3%)."],"supporting_citations":[{"why":"Supplies the GLIGEN architecture used to demonstrate the dataset's effect and the official checkpoint used as the primary comparison baseline.","marker":"[25]"},{"why":"One of the four open-vocabulary detectors that take the LLM flat category list as input and produce detection boxes for ROVI.","marker":"[29]"},{"why":"InstanceDiffusion is the RAM-based tagging pipeline compared against in the data-generation study and a generation baseline in Table 5.","marker":"[51]"},{"why":"InternVL1.5 generates the dense visual descriptions that constitute the pre-detection re-captioning stage.","marker":"[7]"},{"why":"Llama3 performs the two-step extraction and decomposition that turns VLM descriptions into the flat category list for detectors.","marker":"[10]"},{"why":"Qwen2VL runs the yes/no crop-label cross-check used both to filter training boxes and to compute grounding pass rates on generated images.","marker":"[49]"},{"why":"V3Det represents the largest prior detection vocabulary and anchors the claim that ROVI has two orders of magnitude more categories.","marker":"[48]"},{"why":"COYO-700M is one of the large web image-text pools from which ROVI's 1M curated images are sourced.","marker":"[3]"},{"why":"RAM supplies tags in the InstanceDiffusion-style baseline, showing the contrast between recognition-based tagging and re-captioning.","marker":"[58]"}],"fun_headline_variants":["Pre-detection re-captioning boosts instance grounding accuracy","VLM-LLM re-captioning discovers 25K+ object categories","Re-captioning before detection beats state-of-the-art T2I","New re-captioning recipe improves T2I fidelity and aesthetics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Qwen2VL yes/no check reliably tells whether a cropped image matches its label, in both the filtering step that builds ROVI and the evaluation step that reports grounding accuracy; if that model is systematically permissive or biased toward ROVI-style labels, the training labels and the headline pass rates rise together.","fun_headline_variants_meta":{"raw":{"variants":["Pre-detection re-captioning boosts instance grounding accuracy","VLM-LLM re-captioning discovers 25K+ object categories","Re-captioning before detection beats state-of-the-art T2I","New re-captioning recipe improves T2I fidelity and aesthetics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1630,"prompt_tokens":962,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":591}},"tokens_in":578,"tokens_out":668,"duration_ms":7765,"temperature":1.0,"reasoning_tokens":591,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:54:27.099121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both the dataset filtering and the grounding evaluation with an independent verifier, for example human annotators labeling a random sample of the 24,000 inspected boxes or a second, differently trained VLM with a calibrated threshold, and compare acceptance rates between ROVI-generated images and the comparison methods; if the 0.880 vs 0.750-style gaps shrink or reverse, the reported grounding advantage is an artifact of the shared Qwen2VL oracle.","supporting_citations":[{"cited_title":"Gligen: Open-set grounded text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the GLIGEN architecture used to demonstrate the dataset's effect and the official checkpoint used as the primary comparison baseline."},{"cited_title":"Instancediffusion: Instance-level control for image generation, 2024","cited_arxiv_id":null,"evidence_quote":"InstanceDiffusion is the RAM-based tagging pipeline compared against in the data-generation study and a generation baseline in Table 5."},{"cited_title":"V3det: Vast vocabulary visual detection dataset","cited_arxiv_id":null,"evidence_quote":"V3Det represents the largest prior detection vocabulary and anchors the claim that ROVI has two orders of magnitude more categories."},{"cited_title":"Coyo-700m: Image-text pair dataset","cited_arxiv_id":null,"evidence_quote":"COYO-700M is one of the large web image-text pools from which ROVI's 1M curated images are sourced."},{"cited_title":"Recognize anything: A strong image tagging model","cited_arxiv_id":null,"evidence_quote":"RAM supplies tags in the InstanceDiffusion-style baseline, showing the contrast between recognition-based tagging and re-captioning."}],"review_version":1}