{"id":"4ae3e2d5-0670-4a3a-913f-3938a2dd11ac","arxiv_id":"2509.00177","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Category-level retrieval improves when text queries are converted into multiple generated images, aggregated with a learned attention module, and fused with CLIP text similarity.","lead":"This paper improves text-to-image retrieval for category queries by generating several images from the text prompt with a diffusion model and comparing those generated images to database images using a vision model. On 15 benchmarks, the combined text-and-image score beats text-only CLIP and SigLIP retrieval on average, with a few per-dataset exceptions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generated-query fidelity is the load-bearing premise; the paper's own Table 4 shows classes where it fails, so 'consistently outperforms' is not supported at per-class or per-dataset level.","rationale":"The reader's weakest_assumption identifies the generated-query fidelity as the load-bearing premise, and the paper's own evidence supports this. The abstract's 'consistently outperforms' is stronger than what Table 1 shows: for SigLIP, Ours loses on three datasets, and Table 4 shows per-class failures. Since the method's only source of visual information is the GDM, the entire gain hinges on the generator's ability to produce class-representative images in DINOv2 space. The supplementary's real-vs-synthetic query gap quantifies this bottleneck. A per-class statistical analysis would settle whether the failures are rare enough to justify 'consistently' or frequent enough to require caveat. We did not find a more fundamental flaw; the training procedure's λ trick is underspecified but not demonstrably incorrect without code. Thus the reader's CONDITIONAL verdict remains appropriate, but the condition should be made explicit: the claim holds only for classes where the GDM captures the discriminative cues.","tokens_in":23730,"tokens_out":13472,"duration_ms":152269,"concrete_test":"Run the released code to compute per-class mAP for Ours and the text-only baseline on all 15 datasets. Perform a paired Wilcoxon signed-rank test on per-class differences. If the median difference is not significantly >0, or if >5% of classes show a negative difference, the 'consistently outperforms' claim is unsupported. Additionally, for classes with negative differences, compute the DINOv2 similarity between generated queries and real same-class vs. distractor images; if distractor similarity is higher, the mechanism is confirmed as the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'consistently outperforms retrieval methods relying solely on text queries'—rests on the premise that diffusion-generated image queries, aggregated in DINOv2 space, rank true-class database images above distractors. The paper's own analysis (Section 5.5, Table 4) documents failure cases where the generator misses key cues (e.g., hammerhead shark's head shape), and the hybrid score becomes worse than text-only. This is not an edge case: in Table 1, Ours (S,D) is below text-only SigLIP on Stanford Cars (88.2 vs 89.2), Food (92.9 vs 93.1), and FGVC Aircraft (44.1 vs 45.6). The supplementary (Figure 5) shows an 11.6% gap between SD-generated and 'perfect' real visual queries in image-only mAP, indicating the generator is a substantial bottleneck. If the premise fails for a non-trivial fraction of classes, the 'consistently' claim is false and the method's improvement is confined to classes where the GDM happens to be accurate. No variance or per-class significance is reported, so we cannot tell how often the premise fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLETIR, a training-light approach to category-level text-to-image retrieval. Given a text query naming a class, it (1) generates multiple query images with text-to-image diffusion models, (2) encodes them with a vision model (DINOv2) and aggregates the resulting descriptors with a small learned symmetric self-attention network, and (3) fuses the resulting image-to-image similarity with the VLM's text-to-image similarity using a learned scalar lambda. The aggregator and lambda are trained on synthetic images generated from OpenImages class names, with benchmark classes removed. The method is evaluated on 15 datasets under three query types (class name, class description, and both), with CLIP, SigLIP, MetaCLIP, OpenCLIP, and EVA-02-CLIP as VLMs. The main reported result is that on class-name queries the method improves average mAP over text-only CLIP (67.9 vs 62.7) and over text-only SigLIP (72.9 vs 70.3). The paper also reports gains for description-based queries, a small Flickr30k experiment, and robustness results on ImageNet-C.","tokens_in":24105,"tokens_out":6067,"duration_ms":66359,"significance":"If the average improvements are robust, the paper offers a simple and modular way to improve open-vocabulary retrieval by converting text queries into generated visual queries and using a strong vision encoder for intra-modal comparison. A notable strength is that the aggregator and fusion weight are trained on synthetic data with the evaluation benchmarks held out, so the reported retrieval numbers are not fitted. The paper also ships source code and evaluates multiple VLMs and generators, which strengthens reproducibility. The main limitation is that the headline claim of 'consistently outperforming' text-only retrieval is not supported at the per-dataset level, and no uncertainty or per-class analysis is provided. The idea is still valuable if the claim is calibrated to average performance with appropriate statistical support.","major_comments":[{"comment":"The abstract and Section 1 state that the approach 'consistently outperforms' retrieval methods that rely only on text queries. Table 1 does not support this at the per-dataset level. For class-name queries, Ours (S,D) is below text-only SigLIP on Stanford Cars (88.2 vs 89.2), Food (92.9 vs 93.1), and FGVC Aircraft (44.1 vs 45.6). Larger drops appear in the description+class rows, e.g., Stanford Cars 85.5 vs 88.5 and FGVC Aircraft 45.3 vs 48.3. The claim is only supported at the level of the 15-dataset average. Please revise the wording to 'on average' or, preferably, augment the evaluation with per-dataset significance tests and a discussion of the exceptions.","section":"Section 5.2, Table 1"},{"comment":"No variance, confidence intervals, or per-class statistical tests are reported. Several per-dataset differences are small (e.g., Food 92.9 vs 93.1, DTD 50.1 vs 49.9 for CLIP description+class, and Stanford Cars 12.1 vs 12.3 for SigLIP description-only), so the reader cannot tell whether the average improvement is robust or driven by a few large gains. Report bootstrap confidence intervals over classes or datasets, or paired per-class tests, to support the claim that the method improves over the text-only baseline.","section":"Table 1 and Section 5.2"},{"comment":"The paper's own analysis shows that generator fidelity is a load-bearing factor. Supplementary Figure 5 reports an 11.6% mAP gap between SD-generated image queries and 'perfect' real image queries in the image-only setting, and Table 4 documents classes where the generator misses key cues (e.g., hammerhead shark's hammer-shaped head) and the hybrid score degrades. This does not contradict an average-performance claim, but it directly undermines the word 'consistently' in the abstract. I ask for a per-class or per-dataset success/failure analysis: how often does the hybrid score fall below text-only, and is the drop correlated with generation fidelity or with the number of generated images? This would let the reader assess the practical scope of the method.","section":"Section 5.5, Table 4, Supplementary Figure 5"}],"minor_comments":[{"comment":"The definition of the attention layer Al(u) = softmax(phi(u)^T phi(u)) u^T is hard to parse. Please specify the dimensions of u, the softmax axis, and whether the query/key projection is applied per column or per token. The notation u1,i for both input tokens and layer outputs should be disambiguated.","section":"Section 4.2, aggregator equation"},{"comment":"The procedure for removing benchmark classes from OpenImages using CLIP nearest neighbors should be described more precisely. Is a threshold used, or is it a nearest-neighbor match? This matters for reproducibility and for assessing potential label leakage through the filtering step.","section":"Section 4.3"},{"comment":"Fixing the cross-modal similarity of positives to 1 is an ad hoc assumption. Please add an ablation or sensitivity analysis showing the learned lambda values and how the final result changes if this positive-similarity target is varied (e.g., 0.9 or 1.0). This would clarify whether the trick is critical or mostly a stabilizer.","section":"Section 4.3, training trick"},{"comment":"The figure reports results on 14 benchmarks (RESISC45 is excluded). Please state this explicitly in the main text or figure caption, as a reader might otherwise expect 15 datasets.","section":"Supplementary Figure 5"},{"comment":"The column header 'Ground Truth' is slightly misleading; it appears to be a real example image from the class rather than a ground-truth label. Consider renaming it 'Real example' or 'Example database image'.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a clean held-out training protocol, but the headline claim overstates the consistency of the gains. The main revision should focus on recalibrating the abstract and Section 1, and on adding statistical or per-class support. I do not see a load-bearing error that would require rejection; the method appears sound on average. However, if the authors are unwilling to soften the 'consistently outperforms' claim, the paper would not meet the standards of a rigorous retrieval venue. The use of CLIP itself to filter training classes is a minor concern but not fatal given the held-out benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, workmanlike empirical paper. The main idea is to take a text query, generate a handful of images with a text-to-image diffusion model, embed them with DINOv2, aggregate them with a small symmetric-attention module, and fuse that intra-modal similarity with the original VLM text similarity via a learned lambda. The specific combination is new in the retrieval setting—CaFO and SuS-X use generated images for few-shot classification, but not this hybrid retrieval recipe. The evaluation is genuinely broad: 15 datasets, four VLM backbones, class-name/description/both queries, plus a PASCAL VOC comparison. The training protocol is held-out: the aggregator and lambda are fit on synthetic OpenImages classes with benchmark classes removed, so there is no benchmark-result leakage. That matters and the authors deserve credit for it.\n\nThe paper is also honest about its failure modes. Section 5.5 and Table 4 show classes where the generator misses key visual cues and the method falls below text-only. The supplementary's real-vs-synthetic query experiment (11.6% gap in image-only mAP) is a good, self-critical addition.\n\nThe soft spots are in the framing, not the core experiment. The abstract says 'consistently outperforms,' but Table 1 shows per-dataset drops for SigLIP class-name queries on Stanford Cars, Food, and FGVC Aircraft, and a few more drops in the description/combined settings. So 'consistently' is too strong; 'on average' is what the data supports. There are no error bars, no variance across seeds, and no per-dataset significance test, so we can't tell whether the average gains are robust or driven by a few large wins. Also, the formulas in Section 4.2 have a transpose/dimension mismatch: the attention operation as written outputs rows where the text describes columns. A referee should ask for a corrected derivation or a pointer to the code. These are addressable issues, not fatal ones.\n\nBottom line: I would send this to peer review. The empirical contribution is solid and reproducible (code is available), the failure analysis is a plus, and the overclaim is easy to fix. It should be accepted after a revision that tempers the language and adds uncertainty information.","headline":"Sound empirical recipe for improving text-only category retrieval with generated visual queries; average gains are real, but 'consistently outperforms' overstates the per-dataset evidence.","tokens_in":24547,"tokens_out":4244,"would_cite":true,"duration_ms":48438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Turning a category name into generated images beats text-only retrieval on 15 benchmarks.","keywords":["text-to-image retrieval","category-level retrieval","modality gap","diffusion models","vision-language models","DINOv2","zero-shot open-vocabulary retrieval","synthetic query generation"],"falsifier":"Take a class whose generated queries are visibly off, such as hammerhead shark, where the generator omits the hammer-shaped head, and compute the per-class difference between the hybrid mAP and the text-only mAP. If removing the misleading generated image or improving only that class's generation fidelity does not change retrieval, or if classes with poor generation fidelity still improve, then the modality-gap explanation fails. A systematic correlation across all ImageNet classes between generation fidelity (measured, for example, by the CLIPScore of the generated image against the class nam","tokens_in":23688,"feed_emoji":"🖼️","tokens_out":9517,"duration_ms":95814,"temperature":0.7,"pith_summary":"This paper asks whether category-level text-to-image retrieval—finding every image of a named class in a large database—can do better than a text-only vision-language model. The authors argue yes, because text and images occupy separated regions of CLIP-style representation space, while image-to-image comparisons with a self-supervised vision encoder are more reliable. Their recipe is to turn the text query into several images with a text-to-image diffusion model, score database images against those generated images using DINOv2, aggregate the generated queries with a small learned attention network, and blend this intra-modal score with the original text-to-image score. Across fifteen benchmarks, the blended score beats the text-only CLIP and SigLIP baselines for class-name queries, for descriptions that never mention the class name, and for the two combined. The load-bearing premise is that generated images capture enough of each class's visual mode in DINOv2 feature space; when a generator misses a key visual cue, such as the hammer-shaped head of a hammerhead shark, retrieval can get worse.","feed_headline":"Generated image queries beat text-only retrieval on 15 benchmarks","feed_subtitle":"Blending a few generated images with the text score beats CLIP and SigLIP, even for description-only queries.","key_machinery":"The machinery that carries the argument is the aggregator plus the modality split. The aggregator is a sequence of symmetric self-attention layers: query and key projections are shared, value projections are identity, and the CLS token starts as the average of the generated-image features and is the only token updated across layers. Because the values are identities, the output stays in the same DINOv2 feature space as the database images, so the aggregated query can be compared directly to database images. Around this sits the hybrid score s = (1 − λ) sc + λ si, where sc is VLM text-image similarity, si is DINOv2 image-image similarity between the aggregated generated query and the database","core_discovery":"Using a text-only VLM as baseline, the paper demonstrates that replacing pure cross-modal search with a hybrid similarity—one term from text-to-image (CLIP/SigLIP) and one from image-to-image (DINOv2) over diffusion-generated queries—consistently improves category retrieval. With class-name queries, the hybrid reaches 67.9 average mAP with CLIP+DINOv2 versus 62.7 for text-only CLIP, and 72.9 with SigLIP+DINOv2 versus 70.3 for text-only SigLIP, averaged over 15 datasets. The improvement also holds when the query is a class description that omits the class name (+6.4 over CLIP, +3.6 over SigLIP) and when description and name are joined. The paper attributes the gain to replacing cross-modal co","pith_inferences":["Because the method works by translating the query into the visual domain rather than by aligning text and image more tightly, I infer the same recipe should transfer to other cross-modal retrieval tasks—audio-to-image, sketch-to-image, or video retrieval—wherever a generative model can produce a query in the target modality.","The global scalar λ learned once may leave per-class gains on the table; a query-specific or class-specific λ could adapt to how reliable the generated images are for that class, which the paper's failure cases (e.g., hammerhead shark) suggest would help.","I infer that the aggregator could be trained iteratively with the generator: if a diffusion model is fine-tuned to produce queries that maximize the aggregator's retrieval rank, rather than just visual realism, the modality-gap bypass could improve further.","The description-only benchmark built with an LLM opens a measurable new task, but its quality depends on the description generator; as LLMs improve, so should this retrieval setting."],"forward_implications":["Any text-only open-vocabulary retrieval pipeline can be upgraded without retraining the VLM or vision encoder; only the small aggregator and λ are trained, on synthetic images.","Retrieval quality is bounded by the generator: the paper's upper-bound experiment with real query images shows an average gain of about 11.3 mAP over the text-only baseline, so better diffusion models should translate directly into better retrieval.","Class-description-only retrieval becomes practical, letting users find images by appearance or properties when they do not know the category name.","Using several complementary generators together (SD, SD-Turbo, FLUX) yields more gains than using one generator, and the method continues to improve as more query images are added.","The hybrid is more resistant to image corruption than the text-only baseline, with a smaller drop in mAP as ImageNet-C severity increases."],"supporting_citations":[{"why":"Supplies the CLIP text and vision encoders used for the cross-modal baseline and the text-query similarity term sc.","marker":"[48]"},{"why":"Supplies SigLIP, the second vision-language model whose text-only baseline is compared and beaten.","marker":"[77]"},{"why":"Supplies DINOv2, the frozen vision encoder h used for all image-to-image similarity estimation.","marker":"[44]"},{"why":"Supplies Stable Diffusion, the main text-to-image generator for query images and the synthetic training set.","marker":"[50]"},{"why":"Supplies FLUX, the second generator whose images add training and test diversity.","marker":"[33]"},{"why":"Supplies the OpenImages class-name corpus from which the synthetic training set is sampled.","marker":"[32]"},{"why":"Documents the modality gap between text and image representations that motivates converting text queries into images.","marker":"[37]"},{"why":"Provides the prior evidence that intra-modal retrieval is more reliable than cross-modal retrieval, motivating the hybrid.","marker":"[27]"},{"why":"Provides the earlier cascade of generation, vision-model features, and VLM inference that inspires the proposed query-image pipeline.","marker":"[79]"},{"why":"Provides the Tip-Adapter baseline that also combines text and generated image cues; the proposed aggregator is compared against it.","marker":"[78]"}],"fun_headline_variants":["Diffusion-generated queries beat text-only for retrieval","Turning text into images improves category retrieval","Visual queries from diffusion beat text-only retrieval","Hybrid text-image similarity surpasses text-only retrieval"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole gain rests on generated images being close enough to real category appearance in DINOv2 feature space; when a generator drops a class's defining visual cue, the hybrid score can rank distractors above true images and underperform text-only retrieval.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion-generated queries beat text-only for retrieval","Turning text into images improves category retrieval","Visual queries from diffusion beat text-only retrieval","Hybrid text-image similarity surpasses text-only retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2737,"prompt_tokens":725,"completion_tokens":2012,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":1955}},"tokens_in":469,"tokens_out":2012,"duration_ms":15574,"temperature":1.0,"reasoning_tokens":1955,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:52:43.249422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a class whose generated queries are visibly off, such as hammerhead shark, where the generator omits the hammer-shaped head, and compute the per-class difference between the hybrid mAP and the text-only mAP. If removing the misleading generated image or improving only that class's generation fidelity does not change retrieval, or if classes with poor generation fidelity still improve, then the modality-gap explanation fails. A systematic correlation across all ImageNet classes between generation fidelity (measured, for example, by the CLIPScore of the generated image against the class nam","supporting_citations":[{"cited_title":"Learning transferable visual models from natural lan- guage supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP text and vision encoders used for the cross-modal baseline and the text-query similarity term sc."},{"cited_title":"Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski","cited_arxiv_id":null,"evidence_quote":"Supplies DINOv2, the frozen vision encoder h used for all image-to-image similarity estimation."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies Stable Diffusion, the main text-to-image generator for query images and the synthetic training set."},{"cited_title":"Flux1 dev","cited_arxiv_id":null,"evidence_quote":"Supplies FLUX, the second generator whose images add training and test diversity."},{"cited_title":"The open images dataset v4: Unified image classifica- tion, object detection, and visual relationship detection at scale","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenImages class-name corpus from which the synthetic training set is sampled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the modality gap between text and image representations that motivates converting text queries into images."},{"cited_title":"Retrieval-enhanced contrastive vision-text models","cited_arxiv_id":null,"evidence_quote":"Provides the prior evidence that intra-modal retrieval is more reliable than cross-modal retrieval, motivating the hybrid."},{"cited_title":"Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners","cited_arxiv_id":null,"evidence_quote":"Provides the earlier cascade of generation, vision-model features, and VLM inference that inspires the proposed query-image pipeline."}],"review_version":1}