{"id":"8b4c840e-3bfb-400a-8100-cb062353ca7e","arxiv_id":"2412.13834","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CroQS is a new benchmark and task formulation for cross-modal query suggestion, where a system takes an initial text query and a retrieved image cluster and outputs a refined query; captioning and LLM baselines beat the original query on cluster specificity and mAP but trail human suggestions.","lead":"This paper defines a new search task: given a text query and the images it retrieves, suggest refined text queries that each capture a visually consistent subgroup of results. It introduces CroQS, a 50-query benchmark with 295 human-annotated image clusters, and shows captioning and LLM baselines improve cluster-specific recall and representativeness over the original query.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation metrics use the same CLIP model that defines clusters and drives baselines, so the headline gains may measure CLIP self-consistency; a check with independent embeddings or human ratings is needed.","rationale":"The reader's conditional verdict points to the CLIP self-evaluation issue, and that is exactly the most load-bearing concern. The paper's central contribution is a dataset with human-validated clusters and human-written suggested queries, which is real and reusable; the methods and tables are internally consistent. However, the quantitative evidence for the task's usefulness and for the baseline improvements depends on metrics that are computed in the same CLIP space used to create the clusters and to drive the main baselines. This makes the reported gains potentially circular as evidence of semantic quality. The concern is concrete and testable: swapping the evaluation encoder or adding human relevance judgments on a subset would settle whether the benchmark rankings are stable or are an artifact of the evaluation loop. Since the concern does not invalidate the dataset itself, keeping the verdict CONDITIONAL (i.e., no change from the reader's verdict) is appropriate; the condition should include the independent-embedding or human-rating check.","tokens_in":10664,"tokens_out":3180,"duration_ms":32952,"concrete_test":"Recompute Table 1's Cluster Specificity (Eq. 3) and Representativeness (Eq. 4, NDCG, MAP) with a different pretrained text-image encoder (e.g., SigLIP or CLIP ViT-L/14), keeping CroQS clusters and human queries fixed. In addition, collect human ratings of cluster specificity for a random subset of ~50 clusters, and compare the method ranking under CLIP-based RecallCluster with the human ranking. If the relative ordering of ClipCap/DeCap versus GroupCap, or the >115% and >52% improvements over q0, do not survive the encoder swap or disagree with human ratings, the headline measurements are artifacts of the CLIP-based evaluation loop.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CroQS measures cross-modal query suggestion and that the baselines improve over q0—rests on the validity of the metrics in Sec. 4.2, but CLIP appears three times in the loop: it retrieves R(q0, I) and forms the clusters (Sec. 3.2), it supplies the prototype points for captioning baselines (Sec. 5.1), and it scores Cluster Specificity and Representativeness (Eqs. 3-4). Because clusters are defined by CLIP visual-feature proximity, a suggested query is rewarded for matching the CLIP cluster centroid, not necessarily for matching human semantic grouping. The human annotations define the reference suggested queries but are not used to score the metrics, so the reported >115% RecallCluster and >52% Representativeness mAP gains over q0 could largely reflect CLIP self-consistency. This is not an internal inconsistency, but it is a construct-validity gap: if CLIP's embedding geometry misaligns with human judgments on these 50 queries, the paper's quantitative conclusions do not establish that the suggestions are better for users. The benchmark dataset itself remains valuable and reusable, but the headline evaluation needs external validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes a new task, cross-modal query suggestion for text-to-image retrieval, in which a system takes an initial textual query q0 and the visual result set R(q0, I), partitions it into semantic clusters, and generates one suggested query per cluster. The authors contribute CroQS, a benchmark of 50 initial queries with 295 human-validated clusters and human-annotated reference suggestions built on COCO train images. They define three evaluation properties (cluster specificity, representativeness, and similarity to the original query), propose corresponding metrics, and adapt two families of baselines: prototype-captioning methods (ClipCap, DeCap, ClipCapq0) and an LLM-based caption-summarization method (GroupCap). Their experiments report that the baselines improve cluster-specificity recall and representativeness mAP over the initial query, while remaining below human performance on most metrics. The dataset, baseline code, and notebooks are publicly released.","tokens_in":10921,"tokens_out":6439,"duration_ms":61238,"significance":"If the evaluation is valid, the paper makes a useful contribution: it defines a new interactive-retrieval task, provides a reusable human-annotated benchmark, and supplies transparent baselines with standard deviations and open resources. The benchmark itself is potentially valuable independently of the headline numbers, since it provides a shared protocol for future work. However, the central quantitative claims rest on CLIP-based metrics that are entangled with the same model used to build clusters and drive the baselines. Until that construct-validity gap is addressed, the reported improvements over q0 should be treated as provisional. The abstract's paired improvement claim is also not fully supported by the tables as written. These issues are fixable with additional validation experiments and a more careful statement of results, so the paper merits a major revision rather than rejection.","major_comments":[{"comment":"CLIP appears in three places in the evaluation loop: it produces the initial result set and the clusters in §3.2, it supplies the prototype points and representative-image selection for the baselines in §5.1 (Eqs. 5–6), and it scores cluster specificity and representativeness in §4.2 (Eqs. 3–4). Because clusters are subsets of a CLIP-based ranking, a suggested query that moves CLIP rankings toward the CLIP cluster centroid will receive high scores even if it does not align with human semantic grouping; the human-annotated reference suggestions in CroQS are scored with the same CLIP metrics rather than used as an external check. The headline gains in the Abstract (>115% RecallCluster and >52% mAP) may therefore partly reflect within-model self-consistency. Please add a validation study using an independent image-text embedding model and/or human ratings on a sample of clusters, and report the resulting metric values or agreement scores, or explicitly qualify the quantitative conclusions.","section":"§3.2, §4.2 (Eqs. 3–5), §5.1"},{"comment":"The Abstract states that both LLM-based and captioning-based methods improve cluster-specificity recall by more than 115% and representativeness mAP by more than 52% with respect to the initial query. Table 1 shows that no adapted method achieves both improvements simultaneously: ClipCap and DeCap improve RecallCluster by roughly +184% and +179% respectively but mAP by only about +33%, while GroupCap improves mAP by about +52% but RecallCluster by about +105%. Please rephrase the claim so that the metric improvements are attributed to the specific configurations that achieve them, or present paired results that support the current wording. As written, the central quantitative claim in the Abstract is not supported by the reported tables.","section":"Abstract, Table 1"},{"comment":"Eq. (4) defines Recall as |Ci ∩ R(q̂_i, I)| / |Ci| over the whole collection, but §5.2 states that representativeness scores are computed on the top-100 documents. The equation needs an explicit truncation parameter, e.g., R_{100}(q̂_i, I); otherwise the metric as defined differs from the metric that is reported, and for any cluster with |Ci| > 100 the denominator must be reconciled with the cutoff. Please also clarify whether Eq. (3)'s retrieval set R(q0, I) is truncated and how the top-k cutoff k=|Ci| interacts with that truncation. This is not purely notational: the reported representativeness numbers depend on the cutoff.","section":"§4.2, Eq. (4); §5.2"},{"comment":"GroupCap's configuration is incompletely specified: the number k of representative images whose captions are fed to the LLM, the number of few-shot examples in the prompt, the exact prompt template, and the sampling/decoding parameters for Mistral-7B and Llama3-8B are not reported. Without these details, the baseline scores in Tables 1 and 3 cannot be reproduced from the paper text. Please provide a full configuration in an appendix or in the released code documentation, and state which settings were used for the results reported in each table.","section":"§5.1, Figure 2b, Table 3"}],"minor_comments":[{"comment":"The 'CLIP Query' column is not defined in the metric section; please state that it is the cosine similarity between the CLIP text embeddings of q0 and q̂_i, and specify which CLIP model variant and text encoder (e.g., ViT-B/32, which layer) was used.","section":"§4.2, Tables 1–3"},{"comment":"The notation R(q0, I) is used both for the initial result set and for the ranked list produced by a suggested query; please introduce separate notation for a ranked list and for its top-k truncation, as the current overloading makes the formulas harder to follow.","section":"§4.2, Eqs. (3)–(4)"},{"comment":"The sentence in §5.3 that GroupCapLLama3 obtains a 'slightly better macro-averaged score than the human-annotated suggestions' refers only to the similarity-to-q0 property, while §6 says the methods are 'relatively far from human performance'. Please add a qualifier so that the two statements do not appear contradictory.","section":"§5.3, §6"},{"comment":"The label 'LLama30.90' appears to be a typesetting error; the writing should be 'GroupCapLLama3' with the score properly separated. The text also alternates between 'Llama3' and 'LLama3'; please standardize the capitalization.","section":"Table 3"},{"comment":"The paper should state the license and terms of use for the released benchmark, baseline code, and notebooks, and should clarify that COCO's license permits redistribution of the derived cluster/query annotations.","section":"§4.1 and project page"}],"recommendation":"major_revision","confidential_remarks":"The benchmark dataset and open resources are a solid contribution, and the central construct-validity concern is addressable with additional experiments rather than being a fundamental flaw. The abstract overclaim about paired improvements should be corrected regardless of the outcome of those experiments. I do not see a novelty or attribution problem: related work on visual query suggestion is cited and differentiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper. The core contribution is the benchmark. CroQS gives 50 queries, 295 human-curated clusters, and human-annotated suggested queries. That is a real resource for a task that currently has no standard evaluation, and it is reusable even if you never use their baselines. The task formalization—conditioning on an initial query and clusters of unannotated images—is a genuine gap relative to visual query suggestion and group captioning. The paper also ships the dataset, code, and notebooks, which is the right way to do this.\n\nThe baselines are straightforward adaptations, not novel methods, and the paper is honest about that. Tables are internally consistent, standard deviations are reported, and the human upper-bound comparison is useful.\n\nNow the soft spots. The stress-test concern is real but needs to be stated fairly. CLIP appears in cluster construction, prototype selection, and the metrics. Because clusters were created with CLIP features, evaluating suggestions by CLIP retrieval can reward text that hovers near the CLIP centroid even if it would not strike a user as the best label. Human annotations define the groups, but they are not used to score the metrics. So the headline gains over q0 are partly a CLIP self-consistency result. This does not sink the benchmark—it is a construct-validity gap. An easy fix is to re-score the human and baseline suggestions with a different embedding or with human relevance judgments on a subset. Until that check exists, the quantitative conclusions should be read cautiously.\n\nThe abstract overstates the results. More than 115% RecallCluster is true for several baselines, but only GroupCap clears 52% representativeness mAP; DeCap and ClipCap are around 33%. And representativeness Recall for every baseline is below q0's 0.52. The trade-off is plausible—specificity vs. recall—but the abstract should say which method and which metric.\n\nSmaller issues: Eq. 4 does not include the top-100 cutoff mentioned in the text, NDCG@10 is stated but not defined, GroupCap's prompt and number of captioned images are not given, and there is no inter-annotator agreement for the human clusters or suggestions. These are fixable in revision.\n\nWho is this for? Anyone working on interactive text-to-image retrieval, query reformulation, or benchmark design. It deserves a serious referee. I would send it out, ask for the metric definitions and the CLIP validation, and accept after those are addressed.","headline":"A genuinely useful benchmark for a new task, with an evaluation that leans too hard on CLIP and an abstract that overstates the baselines' gains; worth reviewing after tightening.","tokens_in":11433,"tokens_out":2951,"would_cite":true,"duration_ms":28331,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper formalizes cross-modal query suggestion as a task, contributes the CroQS benchmark of 50 queries and 295 human-validated clusters, and shows that captioning and LLM baselines far outperform the initial query on cluster…","keywords":["cross-modal query suggestion","text-to-image retrieval","query reformulation","benchmark","image group captioning","CLIP","large language models","information retrieval"],"falsifier":"Take a held-out set of CroQS clusters and have human annotators rank candidate suggested queries by how well each captures the cluster; compare those rankings with the CLIP-similarity ordering used for the metrics. A large disagreement would show that the benchmark's numbers measure the embedding model rather than the quality of the suggestions.","tokens_in":10497,"feed_emoji":"🔍","tokens_out":5793,"duration_ms":49456,"temperature":0.7,"pith_summary":"This paper establishes cross-modal query suggestion as a measurable retrieval task: starting from a text query and the images it retrieves, a system should propose several refined textual queries, each aimed at one visually coherent subset of the results. To make the task testable, the paper builds CroQS, a benchmark of 50 initial queries with 295 human-validated image clusters and a human-written suggested query per cluster. It defines three evaluation axes—cluster specificity, representativeness, and similarity to the original query—and adapts image-captioning and LLM-based baselines to the task. On the benchmark, those baselines improve cluster-specificity recall by more than 115% and representativeness mean average precision by more than 52% relative to the initial query, while remaining below human-annotated suggestions. If the benchmark holds up, it gives interactive text-to-image search a standard way to compare query-suggestion methods.","feed_headline":"Suggested queries lift image-search recall by 115 percent","feed_subtitle":"A new 50-query benchmark, CroQS, tests whether follow-up queries pinpoint distinct visual groups better than the original wording.","key_machinery":"The load-bearing object is the CroQS benchmark itself: 50 initial queries built on the COCO train split, with 295 human-validated clusters (5.9 clusters per query on average) and one human-annotated suggested query per cluster. The evaluation uses three metrics: RecallCluster, the fraction of a cluster's images ranked in the top $k$ of the initial result set when re-ranked by the suggested query; representativeness measures (Recall, NDCG, MAP) computed by retrieving the cluster's images from the whole collection with the suggested query; and similarity to $q_0$ via Jaccard and CLIP textual similarity. The baseline machinery is prototype captioning—averaging or selecting the most representative CLIP image embedding of a cluster and decoding it with ClipCap or DeCap, optionally conditioned on $q_0$—and GroupCap, which captions the most representative images and asks an LLM to summarize them together with $q_0$.","core_discovery":"The central claim is that query suggestion can be moved from text-only retrieval to cross-modal retrieval, and that the right unit of evaluation is the semantic cluster: a good suggested query should pick out one coherent group of images inside the initial result set without drifting far from the user's original wording. CroQS supplies the ground truth for this claim by fixing the clusters through human judgment, so that different methods can be compared purely on the queries they generate. The paper further claims that off-the-shelf captioning models applied to cluster prototypes, and LLMs that summarize per-image captions, both produce useful suggestions: the captioning-derived methods are the most cluster-specific, the LLM method is the most balanced and stays closest to the initial query, and both beat simply re-using the initial query by large margins. Human annotations still score highest on specificity and representativeness, which the paper reads as evidence that the task is well posed but not yet solved.","pith_inferences":["Beyond the paper, a natural extension is a user study: the benchmark's CLIP-based metrics could be checked against whether human searchers actually prefer the suggested queries when browsing.","Because the clusters are defined in CLIP space, the benchmark's scores are partly a measure of CLIP self-consistency; re-annotating or re-clustering with a different embedding would reveal how much of the reported gain is tied to that specific representation.","The same task definition could transfer to video retrieval or large image archives where captions are absent, since the method only needs embeddings and a captioning model.","CroQS could be extended to a setting where each cluster has multiple equally valid suggested queries, letting metrics reward diversity of good queries rather than a single reference."],"forward_implications":["CroQS can serve as a reusable testbed: any future query-suggestion model can be scored on the same fixed clusters and human-annotated queries without redoing the annotation effort.","Because the benchmark fixes clusters, the comparison isolates query generation from clustering, so method improvements can be attributed to the suggestion module rather than to different groupings.","The gains over the initial query show that even simple captioning and LLM baselines make interactive exploration of image collections feasible in practice.","The gap to human performance identifies a concrete target: methods that keep the specificity of captioning models while matching the query fidelity of LLMs."],"supporting_citations":[{"why":"Supplies the shared image-text embedding space used to cluster images, compute prototypes, and score query-image similarity in the metrics.","marker":"[21]"},{"why":"Provides the COCO train split from which the 50 queries' result sets and 295 annotated image clusters are drawn.","marker":"[12]"},{"why":"ClipCap is one of the adapted captioning baselines; its query-aware variant is tested as ClipCapq0.","marker":"[17]"},{"why":"DeCap is the other prototype-captioning baseline and provides captions inside GroupCap.","marker":"[10]"},{"why":"Llama 3 is the LLM backbone of the best GroupCap configuration.","marker":"[6]"},{"why":"Mistral-7B is the alternative LLM backbone in the GroupCap ablation.","marker":"[8]"},{"why":"Closest prior visual query suggestion system; the paper contrasts its reliance on image captions with the no-metadata setting.","marker":"[29]"},{"why":"Context-aware group captioning with contrastive features, a related method the task extends by conditioning on the initial query.","marker":"[11]"}],"fun_headline_variants":["Cross-modal query suggestions boost image-search recall by 115%","New benchmark CroQS sharpens image-search query suggestions","LLM and captioning models craft better image-search queries","Query suggestion for image search: 115% recall gain via CroQS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that CLIP's text-image similarity matches the way humans group images; CLIP is used both to form the groups and to score the suggestions, so a mismatch would make the reported gains partly an artifact of the system agreeing with itself.","fun_headline_variants_meta":{"raw":{"variants":["Cross-modal query suggestions boost image-search recall by 115%","New benchmark CroQS sharpens image-search query suggestions","LLM and captioning models craft better image-search queries","Query suggestion for image search: 115% recall gain via CroQS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2397,"prompt_tokens":995,"completion_tokens":1402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1341}},"tokens_in":611,"tokens_out":1402,"duration_ms":9637,"temperature":1.0,"reasoning_tokens":1341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:44:47.276795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of CroQS clusters and have human annotators rank candidate suggested queries by how well each captures the cluster; compare those rankings with the CLIP-similarity ordering used for the metrics. A large disagreement would show that the benchmark's numbers measure the embedding model rather than the quality of the suggestions.","supporting_citations":[{"cited_title":"Pacini et al","cited_arxiv_id":null,"evidence_quote":"Provides the COCO train split from which the 50 queries' result sets and 295 annotated image clusters are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ClipCap is one of the adapted captioning baselines; its query-aware variant is tested as ClipCapq0."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeCap is the other prototype-captioning baseline and provides captions inside GroupCap."},{"cited_title":"In: Proceedings of the 17th ACM international conference on Multimedia","cited_arxiv_id":null,"evidence_quote":"Closest prior visual query suggestion system; the paper contrasts its reliance on image captions with the no-metadata setting."},{"cited_title":"In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Context-aware group captioning with contrastive features, a related method the task extends by conditioning on the initial query."}],"review_version":1}