{"id":"e603409c-353e-4cf4-a83b-dd979123f672","arxiv_id":"2504.13172","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SemCORE improves generative cross-modal retrieval by replacing numeric image identifiers with natural-language keywords and adding an MLLM-based semantic verification reranking stage.","lead":"A new retrieval system, SemCORE, makes a multimodal AI model directly generate short text labels (identifiers) for images and then verify its top choices by looking at the images. The paper reports big jumps in text-to-image retrieval accuracy on Flickr30K and MS-COCO, though the gains are partly driven by the verification step and lack some key comparison baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 8.65-point gain over AVG is driven mainly by the GSV reranker, not by the generative SID: removing GSV drops Flickr30K R@1 from 69.0 to 51.9 (Table 3), below AVG's 62.8. Without an AVG-or-CLIP-plus-GSV baseline, the generative-retrieval claim is untested.","rationale":"I read the paper in good faith and find the method coherent and the ablations informative. However, the ablations themselves expose the confound: GSV contributes 17.1 R@1 points on Flickr30K, more than twice the total gap over AVG, and without GSV the method underperforms the strongest generative baseline. Since GSV is a verification/reranking stage over candidate images, it can be applied on top of any first-stage retriever; the paper does not test that alternative, so the central claim about generative retrieval is underdetermined. The reader's stated weakest assumption about compositional generalization of unseen keyword tokens is plausible but secondary; the first-order issue is experimental isolation. The image-to-text comparison is also thin, with only one reproduced generative baseline and no same-backbone comparison. This is a correctable experimental gap, not an internal inconsistency, so the appropriate verdict remains conditional on the missing baselines, matching the reader's CONDITIONAL decision. I therefore set verdict_should_be to UNCHANGED and agreement_with_reader to partial: the reader's rationale mentions the reranker confound, though the formal weakest_assumption field points elsewhere.","tokens_in":15653,"tokens_out":4237,"duration_ms":37600,"concrete_test":"Run AVG (or CLIP) as the first-stage retriever, take its top-10 candidates on Flickr30K and MS-COCO, apply the exactly same GSV verification prompt from Section 3.3, and compare R@1 with SemCORE's reported 69.0 and 42.4. If AVG+GSV or CLIP+GSV reaches or exceeds SemCORE, the reported improvement is attributable to the reranker rather than to the generative SID; if it falls well short, the generative component carries the gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive weakness is experimental isolation of the generative component. In Table 3 (Flickr30K), removing GSV drops R@1 from 69.0 to 51.9, while Table 1 shows AVG at 62.8; so without the verification stage SemCORE is below the best generative baseline by 10.9 points. Yet GSV is not a generative-retrieval operation: it takes a candidate set, constructs a prompt with candidate images and lexical IDs, and asks the MLLM to pick the best match (Section 3.3). Such a reranker can be grafted onto any first-stage retriever. The paper never reports AVG+GSV or CLIP-top-k+GSV; therefore the headline '8.65 average R@1 improvement' may reflect the MLLM verification strength, not SID or generative decoding. The 'w/o SID' ablation (66.1 vs 69.0) shows SID contributes only modestly. The image-to-text comparison also uses only one reproduced generative baseline (GRACE*) and omits AVG as 'unsuitable', leaving no same-backbone generative comparison. The central claim that SemCORE advances generative cross-modal retrieval is thus not yet isolated from a standard retrieve-and-rerank explanation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SemCORE proposes a unified generative cross-modal retrieval framework in which each image is assigned a Structured natural language IDentifier (SID), formed by a K-Means-based global ID and a KeyBERT-based lexical ID, and an MLLM is fine-tuned to generate these identifiers under constrained decoding. A Generative Semantic Verification (GSV) stage then reranks a small candidate set by prompting the MLLM to select the image that best matches the query. The paper reports text-to-image Recall@1 gains over prior generative baselines on Flickr30K and MS-COCO, with an average improvement of 8.65 points, and reports image-to-text results on Flickr30K that match or exceed traditional one-tower and two-tower methods. The authors claim that SemCORE is the first generative cross-modal retrieval framework to handle both text-to-image and image-to-text retrieval.","tokens_in":16090,"tokens_out":3922,"duration_ms":35387,"significance":"If the reported gains are attributable to the proposed identifier design and generative decoding, the paper would make a useful contribution by showing that generative cross-modal retrieval can be competitive with similarity-based retrieval while unifying both retrieval directions. The paper is clearly structured, presents extensive ablations, includes a scaling study across backbone sizes, and provides concrete case analyses. However, the current experiments do not isolate the effect of the generative component from the effect of the MLLM-based verification reranker, and the comparison is confounded by differences in backbone models. These issues are load-bearing for the central claim, so the significance cannot be fully assessed until the missing baselines and ablations are supplied.","major_comments":[{"comment":"The decisive ablation is missing. In Table 3, removing GSV drops Flickr30K R@1 from 69.0 to 51.9, while Table 1 reports the best generative baseline AVG at 62.8; without the verification stage, SemCORE is 10.9 points below AVG. Because GSV (Section 3.3) is a candidate reranker that can be attached to any first-stage retriever, the headline '8.65-point average R@1 improvement' is not yet shown to be an advance in generative retrieval. Please report AVG+GSV and a CLIP-top-k+GSV baseline, and report the full SemCORE pipeline both with and without the reranker.","section":"Table 3 / Table 1"},{"comment":"The comparison is confounded by backbone differences. SemCORE uses InternVL2.5 (Section 3.4), while the generative baselines GRACE, IRGen, and AVG use different base models, so the reported gains may reflect the capacity of the base MLLM rather than the proposed SID and GSV. Please add a same-backbone generative baseline, such as GRACE or IRGen fine-tuned on InternVL2.5, or an equivalently capable generative baseline, to establish that the gains come from the identifier design and decoding strategy.","section":"Section 4.1.2 / Table 1"},{"comment":"The identifier construction is self-referential. Lexical IDs are extracted from captions generated by an MLLM (Section 3.2.1), the same family of MLLMs is fine-tuned to generate those IDs (Section 3.4), and the same model family is used in the GSV verification prompt (Section 3.3). The reported 'semantic understanding' gains may therefore partly measure the model's agreement with its own captioning behavior. Please evaluate identifiers built from human captions or from an off-the-shelf captioning model of a different family, and report whether test-set keyword tokens that the model never saw during training are actually generated for unseen images.","section":"Section 3.2 / Section 3.3"},{"comment":"The image-to-text evaluation is not yet sufficient to support the claim that SemCORE bridges the gap between generative and traditional cross-modal retrieval. Only one generative baseline (GRACE*) is reproduced, and AVG is excluded because it is not based on an MLLM (Section 4.1.2). Please compare against a strong generative MLLM baseline, ideally with the same backbone as SemCORE, and report image-to-text results for that baseline both with and without a GSV-style reranker.","section":"Table 2 / Section 4.2.2"}],"minor_comments":[{"comment":"The table heading 'w/o GCV' should be 'w/o GSV' to match the text, and the text contains a typo 'SemCOER' in the ablation paragraph.","section":"Table 3 / Section 4.3"},{"comment":"The word 'framweork' in the first sentence of Section 4.2 is a typo and should be 'framework'.","section":"Section 4.2"},{"comment":"The appendix states that the GSV candidate set size is fixed at 10, but Section 3.3 does not specify how the candidate set is selected from the beam output or how the candidate set relates to the beam size; please clarify this procedure.","section":"Appendix A.1 / Section 3.3"},{"comment":"The figure caption does not name the axes explicitly; please state in the caption that the horizontal axis is cluster size and the vertical axis is Recall@1, or add axis labels to the figure itself.","section":"Figure 3"},{"comment":"The metric name is written inconsistently as 'rSum' in the text and 'rsum' in Table 4; please unify the notation.","section":"Tables 1-4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting, but the experimental isolation problem is real and central: the GSV reranker likely explains a large part of the gain, and the absence of any same-backbone generative baseline makes the 'generative retrieval advance' claim hard to evaluate. The authors should also be asked to address the self-referential nature of the identifiers, since this affects how reviewers interpret the semantic-understanding claims. With the requested baselines and analyses, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid engineering paper with a genuinely useful design—structured natural-language identifiers built from cluster prototypes plus KeyBERT keywords, and an MLLM verification stage—but the headline improvement is mostly the reranker, not the generative identifier. The paper deserves peer review, but the central claim needs re-framing and additional baselines before it can be taken at face value.\n\nWhat is actually new: combining a coarse global cluster ID with a lexical keyword ID, both derived from MLLM-generated captions, is a reasonable next step after GRACE and AVG. Making the same framework handle text-to-image and image-to-text is also new for generative cross-modal retrieval, and the image-to-text result (88.8 R@1 on Flickr30K) is genuinely competitive with traditional one-tower and two-tower methods. The ablations are more thorough than usual: w/o GSV, w/o SID, w/o global ID, w/o dedup, w/o constrained decoding, plus beam-size and backbone-scaling studies. The paper is also honest about the R@5 trade-off and the false-negative issue.\n\nThe soft spots are real and they matter. In Table 3, removing GSV drops Flickr30K R@1 from 69.0 to 51.9, below AVG's 62.8. GSV is not a generative-retrieval operation as the field defines it: it takes a candidate set, builds a prompt with images and lexical IDs, and asks the MLLM to pick the best match. That is a reranker, and it could be grafted onto AVG or a CLIP retriever. The paper never reports AVG+GSV or CLIP-top-k+GSV, so we cannot tell whether SID plus generative decoding is what produces the 8.65-point average gain, or whether any decent first stage plus an MLLM reranker would do it. The w/o SID ablation (66.1 vs 69.0) suggests SID itself contributes only a few points. The image-to-text comparison reproduces only GRACE and omits AVG, which weakens the \"first to bridge the gap\" claim.\n\nThere is also a self-reference issue: the captions used to build SIDs come from the same MLLM family that is fine-tuned to generate the IDs and then used to verify candidates. That does not invalidate the results, but it means \"semantic understanding\" is partly the model agreeing with itself, and it deserves a controlled variant, e.g., captions from a different model.\n\nBottom line: this is a useful contribution and should go to peer review. But the authors should be asked to add same-backbone generative baselines, reranker-grafted baselines, and code release. Without those, the generative-retrieval claim is untested.","headline":"SemCORE's structured natural-language IDs plus an MLLM verification reranker is a reasonable engineering contribution, but the headline gains are driven mainly by the reranker, not the generative identifier, and the paper needs extra baselines before the central claim is credible.","tokens_in":16495,"tokens_out":2805,"would_cite":true,"duration_ms":24763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SemCORE claims that giving generative cross-modal retrievers natural-language identifiers—keyword descriptions of each image plus an MLLM verification step—raises text-to-image Recall@1 by 8.65 points on average and matches traditional…","keywords":["cross-modal retrieval","generative retrieval","multi-modal large language models","structured natural language identifier","semantic verification","text-to-image retrieval","image-to-text retrieval","Recall@1"],"falsifier":"Build a Flickr30K split where every test image's lexical ID contains at least one keyword token that never appears in any training identifier, run SemCORE with and without GSV, and compare R@1. If performance falls to near chance without GSV, the claimed compositional generalization collapses; if it stays high, the verification step alone is doing the work.","tokens_in":1544,"feed_emoji":"🖼️","tokens_out":1993,"duration_ms":52806,"temperature":0.7,"pith_summary":"SemCORE claims that generative cross-modal retrieval can be made substantially more accurate if the identifiers a model is trained to emit are built from natural language rather than numbers or cluster labels. The paper proposes a Structured natural language IDentifier (SID): a cluster-based global ID for coarse localization plus keyword-based lexical IDs extracted from MLLM-generated captions, followed by a Generative Semantic Verification (GSV) step in which the MLLM picks the best candidate among a shortlist. On Flickr30K and MS-COCO, the framework reports an average gain of 8.65 points in text-to-image Recall@1 over prior generative methods, and for image-to-text retrieval it reports results on par with or above traditional one-tower and two-tower baselines. The paper also claims to be the first generative cross-modal retrieval framework to handle both retrieval directions.","feed_headline":"Natural-language IDs lift generative image retrieval by 8.65 points","feed_subtitle":"SemCORE pairs caption keywords with an MLLM reranking step and also matches classic image-to-text baselines.","key_machinery":"The load-bearing mechanism is the SID-GSV pipeline. SID is a two-part identifier: a Global ID (a K-Means cluster index over caption embeddings) that narrows the search, and a Lexical ID (four TF-IDF-deduplicated KeyBERT keywords from the image's MLLM-generated caption) that carries fine-grained semantics. GSV then reranks the beam of candidates by prompting the MLLM to compare each candidate image (with its lexical ID) against the query and select the best match. Constrained beam search over a Trie of valid SIDs keeps every emitted identifier legal.","core_discovery":"The central discovery is that identifier semantics, not just the generative backbone, determine how well an MLLM can retrieve. Previous generative retrievers used hierarchical-clustering numeric IDs or tokenized image codes; SemCORE replaces the final random suffix with keywords that describe the image and lets the MLLM verify the top candidates with an image-plus-keywords prompt. The authors argue that this activates the model's pretrained natural-language understanding, turning retrieval into a form of grounded generation, and that the same design works symmetrically for text-to-image and image-to-text.","pith_inferences":["The reported 8.65-point gain likely bundles the effect of the verification reranker with the effect of the semantic identifiers; an ablation that holds GSV fixed while swapping SID for random suffixes would separate the two contributions, and the paper does not report it.","Because identifiers are built from MLLM-generated captions and the same model family is fine-tuned to emit them, the semantic signal is partially self-referential; building identifiers with a captioning model from a different family could test whether the gain survives.","The method's dependence on keyword tokens that appear at test time suggests a compositional-generalization test: construct a split where test identifiers contain only unseen keywords and measure R@1 without GSV to see whether the model is truly generalizing."],"forward_implications":["Text-to-image generative retrieval becomes competitive with traditional similarity-based methods, since Recall@1 improves by 8.65 points on average and rSum is best on both evaluated datasets.","Image-to-text retrieval, previously weak in the generative paradigm, can match or exceed one-tower and two-tower baselines while preserving the efficiency of generative decoding.","Larger MLLM backbones keep improving retrieval performance from 1B to 8B parameters, so the approach should benefit from further model scaling.","Removing SID or GSV hurts results sharply (R@1 drops from 69.0 to 66.1 with SID removed and to 51.9 with GSV removed), showing that both semantic identifiers and verification carry weight."],"supporting_citations":[{"why":"Defines the generative cross-modal retrieval paradigm and the constrained decoding algorithm SemCORE inherits and compares against as the GRACE baseline.","marker":"[21]"},{"why":"Provides the strongest generative baseline (AVG), which SemCORE outperforms and whose token-to-voken formulation motivates the semantic identifier design.","marker":"[20]"},{"why":"Supplies the InternVL2.5 backbone MLLM that SemCORE fine-tunes for both identifier generation and semantic verification.","marker":"[5]"},{"why":"KeyBERT is the keyword extraction technique used to build the lexical ID component of SID.","marker":"[10]"},{"why":"K-Means clustering constructs the global ID that captures macroscopic semantic information.","marker":"[12]"},{"why":"The TF-IDF weighting scheme underpins the semantic deduplication used to keep lexical IDs distinct from the global ID.","marker":"[34]"},{"why":"DSI is the foundational generative retrieval method that establishes identifier construction and memory-as-index, informing the SID design.","marker":"[35]"},{"why":"LeaPRR is the one-tower baseline for image-to-text retrieval that SemCORE claims to match or surpass.","marker":"[28]"},{"why":"FNE serves as a two-tower baseline and also supplies the false-negative analysis used to interpret SemCORE's seemingly incorrect retrievals.","marker":"[18]"}],"fun_headline_variants":["Semantic IDs lift generative retrieval by 8.65 points","Keywords, not codes: SemCORE sharpens multimodal retrieval","Both ways: SemCORE's semantic IDs power retrieval","8.65-point gain: SemCORE's semantic IDs beat numeric codes"],"cache_read_input_tokens":18560,"weakest_assumption_plain":"The framework assumes the model can, at test time, generate keyword tokens for images it has never seen even when those keywords did not appear in any training identifier, relying on semantic similarity rather than memorization, and that this transfer—not the verification step—is what drives the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Semantic IDs lift generative retrieval by 8.65 points","Keywords, not codes: SemCORE sharpens multimodal retrieval","Both ways: SemCORE's semantic IDs power retrieval","8.65-point gain: SemCORE's semantic IDs beat numeric codes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001379,"raw_usage":{"total_tokens":5571,"prompt_tokens":914,"completion_tokens":4657,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":4584}},"tokens_in":530,"tokens_out":4657,"duration_ms":26975,"temperature":1.0,"reasoning_tokens":4584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:12:40.676520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a Flickr30K split where every test image's lexical ID contains at least one keyword token that never appears in any training identifier, run SemCORE with and without GSV, and compare R@1. If performance falls to near chance without GSV, the claimed compositional generalization collapses; if it stays high, the verification step alone is doing the work.","supporting_citations":[{"cited_title":"Revolutionizing Text-to-Image Retrieval as Autoregressive Token-to-Voken Generation","cited_arxiv_id":"2407.17274","evidence_quote":"Provides the strongest generative baseline (AVG), which SemCORE outperforms and whose token-to-voken formulation motivates the semantic identifier design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DSI is the foundational generative retrieval method that establishes identifier construction and memory-as-index, informing the SID design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LeaPRR is the one-tower baseline for image-to-text retrieval that SemCORE claims to match or surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FNE serves as a two-tower baseline and also supplies the false-negative analysis used to interpret SemCORE's seemingly incorrect retrievals."}],"review_version":1}