{"id":"60493e14-4a37-4880-b238-f76010f3a67d","arxiv_id":"2507.19054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GR-CLIP, a mean-centering calibration that removes CLIP's modality gap, improves mixed modality retrieval by up to 26 NDCG@10 points and outperforms a 75x heavier generative embedding model.","lead":"This paper shows that CLIP embeddings have a modality gap that biases retrieval rankings when the search corpus mixes images, text, and combined documents. Subtracting the per-modality mean vector from every embedding before ranking fixes much of the bias at almost no computational cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GR-CLIP's central attribution to modality-gap removal is confounded by an undocumented query-mean subtraction, so an ablation is needed before the causal claim is accepted.","rationale":"The paper's central claim is that mean-centering 'removes the modality gap' and thereby yields large retrieval gains. The derivation in Section 2.3 builds on the constant-gap model eT_i - eI_i = c_perp and suggests subtracting modality-specific means. However, the actual algorithm in Appendix B subtracts a third mean, the query mean, from all queries, explicitly motivated by query-document style differences rather than by the modality gap. The reported experiments therefore do not isolate the effect of modality-gap removal; they test a combined transformation. The paper provides no ablation isolating the query mean. Since the code is publicly available, a direct experiment can settle this quickly. If the query-mean subtraction is responsible for the gains, the headline explanation and the attribution of improvements to gap removal would need to be revised; if not, the concern is resolved. This is a hidden component in the method that directly bears on the causal claim, which is why it is the most load-bearing issue. I keep the CONDITIONAL verdict from the reader and recommend adding this specific condition, along with the other reproducibility items already noted (fusion weight, error bars, simulation circularity).","tokens_in":19563,"tokens_out":12211,"duration_ms":116800,"concrete_test":"Run the MixBench evaluation (all four subsets; CLIP-B/16 and SigLIP-400M) under three conditions: (A) full GR-CLIP as in Algorithm 1; (B) the same pipeline but using the document text mean T for queries as well, removing the query-specific mean; (C) a minimal gap-vector correction that replaces text embeddings by eT_i - (T - I) and leaves image embeddings and queries untouched. Compare NDCG@10 against the CLIP baseline. If condition (B) or (C) retains at least 80% of the full GR-CLIP gain, the modality-gap attribution is supported; if the gain largely collapses, the improvement is driven by the query-mean normalization rather than by closing the image-text gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 in Appendix B subtracts three distinct means: query mean q, document text mean T, and document image mean I. The derivation in Section 2.3 justifies only the subtraction of modality means (T and I) based on the model eT_i - eI_i = c_perp. The query-mean subtraction is an additional step, explicitly described in Appendix B as accounting for 'structural and semantic differences' between short queries and long documents; it is not implied by the modality-gap theory. The main text's claim that 'the entire improvement comes from subtracting modality means' is therefore not directly supported by the reported experiments, because the evaluation includes this extra query-normalization component. If the query-mean subtraction drives most of the gain, the method's success reflects query-document style alignment rather than closing the image-text gap, which would substantially weaken the paper's causal narrative. A secondary inconsistency: the abstract says 'up to 26 percentage points,' but Table 1 shows gains as large as about 0.38 NDCG@10 (e.g., SigLIP-400m on VisualNews), suggesting the headline number needs clarification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies mixed-modality retrieval, in which queries and corpus documents can be text, image, or multimodal. It observes a modality gap in CLIP-style embeddings and proposes GR-CLIP, a post-hoc calibration that subtracts modality-specific mean vectors from embeddings before cosine-similarity ranking. It introduces MixBench, constructed from Google-WIT, MSCOCO, OVEN, and VisualNews, and reports that GR-CLIP improves NDCG@10 over the corresponding CLIP models across five CLIP variants, outperforms VLM2Vec on most subsets, and generalizes to video-text and audio-text settings. The main causal story is that removing the modality gap resolves intra-modal ranking bias and fusion failure.","tokens_in":19750,"tokens_out":7739,"duration_ms":75006,"significance":"If the results hold, the paper makes a useful empirical and practical contribution: it demonstrates that a few mean subtractions applied to frozen CLIP embeddings can produce large gains on heterogeneous retrieval, and it provides a benchmark (MixBench) plus code and data for future work. The internal consistency across five CLIP variants, four datasets, three metrics, and additional audio/video pairs is a genuine strength, as is the reproducible artifact release. The significance would be higher once the attribution of the gains to modality-gap removal rather than query-style normalization is established by ablation, and once the headline numbers are reconciled with the tables.","major_comments":[{"comment":"The main text and the derivation in §2.3 justify GR-CLIP by subtracting only the document text mean and the document image mean, based on the model eT_i - eI_i ≈ c_perp. Algorithm 1 in Appendix B additionally subtracts a query mean e_q (and a separate OVEN-specific query mean), with the stated purpose of accounting for 'structural and semantic differences' between short queries and long documents. The reported MixBench results therefore evaluate a three-mean procedure, not solely the modality-gap removal described in the main text, so the claim that the gains come from closing the modality gap is not directly supported. Please provide an ablation on MixBench comparing (i) image/text document mean subtraction only, (ii) query-mean subtraction only, and (iii) the full procedure; this is needed to attribute the observed improvement.","section":"§2.3; Appendix B, Algorithm 1"},{"comment":"The abstract and §5.2 state an 'up to 26 percentage points' gain, but Table 1 contains larger gaps, e.g., SigLIP-400m on VisualNews goes from 0.385 to 0.769 NDCG@10 (+0.384) and on MSCOCO from 0.327 to 0.696 (+0.369). Please correct the headline, specify the exact subset/condition to which 26 points refers, and state whether the reported values are raw NDCG differences rather than relative improvements.","section":"Abstract; Table 1"},{"comment":"Table 2 reports OVEN queries as T+I (image plus text), and Appendix E states that the OVEN query remains unchanged, but Algorithm 1 and §2.3 describe a pipeline in which queries are text-only and encoded with f_T only. The manuscript never specifies how the image component of an OVEN query is handled for CLIP, GR-CLIP, or VLM2Vec. This omission makes the OVEN rows in Table 1 unreproducible and leaves open the possibility that the comparison treats the two methods asymmetrically. Please clarify the exact query encoding used for OVEN (e.g., text-only, image-only, or fused) for every method.","section":"§4.1; Appendix E; Appendix B Algorithm 1"},{"comment":"The fusion weight α for multimodal documents in the final MixBench results is not reported. Section 2.3 defines the interpolation e = α e_T + (1−α) e_I, and Section 4 shows that performance varies strongly with α (including a U-shape for original CLIP), but Figure 4b and Table 1 do not state which α was used per dataset/model or whether it was tuned. Please report the α values used and any selection criterion, since without this the headline benchmark numbers cannot be reproduced or compared across methods.","section":"§5.2; Figure 4b; §2.3"}],"minor_comments":[{"comment":"The push-down simulation assigns zero similarity to all screenshots, which by construction imposes the modality-bias being tested; please label it as an illustrative mechanism rather than an independent confirmation, or add a less extreme penalty variant.","section":"§3.2"},{"comment":"The second bullet point in the definition of the two distinguishing properties is labeled 'b)' while the first is labeled '1)'; renumber for consistency.","section":"§2.1"},{"comment":"The caption contains the leftover text 'Reviewer #2: This is a very interesting paper! I really like it!' in the top-left panel; this appears to be a template artifact and should be removed.","section":"Figure 1 caption"},{"comment":"The manuscript calls the GitHub and Hugging Face links 'anonymous', but the URLs contain 'yuhui-zh15', which reveals an author identity; if this is a double-blind submission, the links should be anonymized.","section":"Code/Data Availability"},{"comment":"The label 'SigLIP-ViusalNews' is a typo for 'VisualNews'.","section":"Figure 7 caption"},{"comment":"The tables do not report error bars or significance tests; since the evaluation appears deterministic given the fixed query/document sets, state this explicitly and mention whether any randomness (e.g., caption sampling or GPT generation) was involved in the reported numbers.","section":"Tables 1 and Figures 4-7"}],"recommendation":"major_revision","confidential_remarks":"The overall direction is promising and the empirical pattern is consistent, but the attribution of the gains to modality-gap removal needs an ablation, and the OVEN query encoding plus headline-number mismatch are unresolved. I recommend requesting a major revision and asking for the ablation and clarifications before further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a paper worth engaging with, but the central causal claim is not yet supported by the experiments as described. The actual algorithm in Appendix B subtracts three means (query, document text, document image), while the derivation and main-text narrative only motivate subtracting the two modality means. There is no ablation separating the query-mean component, so we cannot tell whether the big NDCG gains come from closing the image-text gap or from aligning short queries to long documents. That is the main issue, and it is fixable.\n\nWhat I credit: the paper defines mixed modality search cleanly and shows a clear U-shaped performance curve as the corpus mixes modalities; MixBench is a useful, reusable evaluation suite; the empirical sweep over five CLIP variants and four datasets is thorough; and the method is almost embarrassingly simple, with code and data released. The gains are large and consistent, so the practical finding—mean-centering helps mixed retrieval—is likely robust.\n\nNow the soft spots, in proportion. The query-mean confound is the big one; an ablation (T/I only, query only, both) is necessary. Second, the abstract says 'up to 26 percentage points' but Table 1 shows SigLIP gains of about 0.38 NDCG@10 on VisualNews and MSCOCO, so the headline is inconsistent with the paper's own table. Third, MixBench results do not report the fusion weight α; with a free parameter like that, readers need to know if the conclusions hold across α. Fourth, there are no error bars or significance tests; at ~1000 queries, per-point differences may be noise. Fifth, the GPT-generated documents on MSCOCO preserve the sampled query caption and VisualNews documents expand on the query caption, which risks near-duplicate text-text matches. That does not invalidate the cross-modal findings but makes the benchmark's text-text mode easier than it looks. Sixth, the push-down simulation encodes the bias it claims to confirm; it is a sanity check, not evidence.\n\nCitation pattern looks fine; the method is positioned properly relative to prior modality-gap work.\n\nFor the review process: send it out. A serious referee should ask for the ablation, the α values, error bars, and a corrected headline. The paper is a solid empirical contribution that currently over-claims its explanation.","headline":"Useful benchmark and a simple method with real gains, but the missing ablation on the query-mean term leaves the modality-gap attribution unproven.","tokens_in":20300,"tokens_out":4516,"would_cite":true,"duration_ms":42582,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GR-CLIP removes CLIP's modality gap and lifts mixed-modality NDCG@10 by up to 26 points.","keywords":["mixed modality search","modality gap","CLIP","embedding calibration","retrieval","NDCG","multimodal fusion","VLM2Vec"],"falsifier":"Compute residuals $r_i = (e^T_i - e^I_i) - (\\bar e^T - \\bar e^I)$ on a held-out corpus; if the residual variance is large or the residuals correlate with semantic content, the constant-offset model fails. Alternatively, estimate means on one document distribution and evaluate on another; if NDCG@10 gains vanish or reverse, the gap is not a domain-independent constant.","tokens_in":19367,"feed_emoji":"🔎","tokens_out":9369,"duration_ms":85309,"temperature":0.7,"pith_summary":"This paper argues that CLIP-style contrastive models are poorly suited to mixed modality search—retrieval over a corpus that mixes text-only, image-only, and image-text documents—because their embeddings carry a modality gap: image and text features cluster separately, so same-modality items are ranked too high and fused image-text embeddings land in weak regions. It proposes GR-CLIP, a post-hoc calibration that subtracts each modality's mean embedding before computing cosine similarity, which cancels the constant offset between modalities. On MixBench, a new benchmark built from four image-text datasets, GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP and outperforms the generative embedding model VLM2Vec by about 4 points while using roughly 75× less compute. The practical stakes are that a nearly free, training-free preprocessing step may capture most of the benefit that expensive generative embedding models provide for heterogeneous retrieval.","feed_headline":"Subtracting two means boosts CLIP retrieval by 26 points","feed_subtitle":"A post-hoc calibration removes CLIP's modality gap, beating generative embeddings with 75x less compute.","key_machinery":"The key machinery is the modality-gap vector and its cancellation by mean-shift calibration. Prior work has characterized the gap as a constant vector $c_\\perp$ such that $e^T_i - e^I_i \\approx c_\\perp$ for paired items, orthogonal to the shared image/text subspace. GR-CLIP estimates $c_\\perp$ as the difference between dataset-level mean text and image embeddings, subtracts each modality's mean from its own embeddings—using a separate mean for short text queries versus long text documents—and then ranks by cosine similarity. The subtraction makes $e'^T_i - e'^I_i \\approx 0$, so cross-modal similarities become comparable to intra-modal ones, and the same calibration is applied inside the linear fusion used for image-text documents.","core_discovery":"The central discovery is that the modality gap—a near-constant offset vector $c_\\perp$ between paired image and text embeddings, approximately orthogonal to the semantic subspace—is the main obstacle preventing CLIP from ranking across mixed-modality corpora. The paper shows that CLIP's retrieval performance traces a U-shaped curve as text documents are replaced by screenshots: performance collapses when most documents are in the other modality, then recovers when all are, a pattern reproduced by a simulation that pushes cross-modal items to the bottom. Removing the gap by subtracting modality-specific means flattens the curve, makes linear fusion of image and text features useful at intermediate fusion weights, and on MixBench raises NDCG@10 by up to 26 points across five CLIP variants, surpassing the VLM2Vec baseline by about 4 points with 75× less compute; the exception is MSCOCO, which VLM2Vec was trained on.","pith_inferences":["If the constant-offset characterization holds in other domains, the same mean-subtraction calibration should be applied to every contrastive embedding system (audio, video, and future modalities) before attributing retrieval gains to heavier models.","Because the paper estimates means on training splits and applies them to test corpora, a natural extension is to estimate means from the unlabeled target corpus itself and measure whether that improves transfer under domain shift.","These results suggest retrieval benchmark reports should include a calibrated contrastive baseline; otherwise, some reported advantages of generative embedding models over CLIP may simply reflect an uncalibrated modality gap.","A direct follow-up is to check whether generative embedding models like VLM2Vec have their own modality gap and whether the same mean subtraction shrinks their gap and their compute disadvantage."],"forward_implications":["On MixBench, GR-CLIP raises NDCG@10 by up to 26 percentage points over the corresponding CLIP variant, with gains on all four subsets and across all five CLIP, OpenCLIP, and SigLIP variants tested.","GR-CLIP outperforms the VLM2Vec generative-embedding baseline by about 4 NDCG@10 points on average while using roughly 75× less inference compute; the MSCOCO subset is the exception because VLM2Vec was trained on it.","The U-shaped retrieval curve observed as text documents are replaced by screenshots is explained by modality-gap ranking bias and is flattened after mean-shift calibration.","After calibration, linear fusion of image and text embeddings peaks at intermediate fusion weights and beats both unimodal endpoints, so multimodal documents contribute positively rather than degrading retrieval.","The same pattern holds under NDCG@100 and Recall@1, and for text-to-video and text-to-audio retrieval."],"supporting_citations":[{"why":"Supplies the constant-offset characterization of the modality gap from which GR-CLIP's mean subtraction is derived.","marker":"[36]"},{"why":"Documents the modality gap's existence and clustering, motivating the ranking-bias and fusion-failure analysis.","marker":"[17]"},{"why":"Defines the CLIP baseline that GR-CLIP calibrates, including the separate image and text encoders and cosine similarity ranking.","marker":"[25]"},{"why":"Provides the strong generative-embedding baseline that GR-CLIP is compared against and that it outperforms with far less compute.","marker":"[12]"},{"why":"Supports the constant-vector approximation of the modality gap used in the method's derivation.","marker":"[35]"},{"why":"Source dataset (Google WIT) for one MixBench subset and for the mean-calibration statistics.","marker":"[27]"},{"why":"Source dataset (MSCOCO) for a MixBench subset and the one where the VLM2Vec baseline was trained.","marker":"[18]"},{"why":"Source dataset (OVEN) that supplies an existing query-to-multimodal-document retrieval format used in MixBench.","marker":"[10]"},{"why":"Source dataset (VisualNews) for a MixBench subset.","marker":"[19]"}],"fun_headline_variants":["Subtract two means, boost CLIP retrieval by 26 points","A simple subtraction closes the modality gap for CLIP","26-point jump: removing CLIP's modality gap","Post-hoc fix: subtract means, beat generative embeddings","75x less compute: calibrate CLIP by subtracting means"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the modality gap is a single constant vector shared by all paired image-text items, so subtracting dataset-level means removes it; if the gap varies with content or has a component inside the semantic subspace, mean subtraction only partially closes it and the claimed gains may fail to transfer to new corpora or domains.","fun_headline_variants_meta":{"raw":{"variants":["Subtract two means, boost CLIP retrieval by 26 points","A simple subtraction closes the modality gap for CLIP","26-point jump: removing CLIP's modality gap","Post-hoc fix: subtract means, beat generative embeddings","75x less compute: calibrate CLIP by subtracting means"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2822,"prompt_tokens":886,"completion_tokens":1936,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1855}},"tokens_in":502,"tokens_out":1936,"duration_ms":14670,"temperature":1.0,"reasoning_tokens":1855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:02:59.028984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute residuals $r_i = (e^T_i - e^I_i) - (\\bar e^T - \\bar e^I)$ on a held-out corpus; if the residual variance is large or the residuals correlate with semantic content, the constant-offset model fails. Alternatively, estimate means on one document distribution and evaluate on another; if NDCG@10 gains vanish or reverse, the gap is not a domain-independent constant.","supporting_citations":[{"cited_title":"Retrieve a relevant item that represents: {Query}\\n","cited_arxiv_id":null,"evidence_quote":"Supplies the constant-offset characterization of the modality gap from which GR-CLIP's mean subtraction is derived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the modality gap's existence and clustering, motivating the ranking-bias and fusion-failure analysis."},{"cited_title":"Jiang, R","cited_arxiv_id":null,"evidence_quote":"Provides the strong generative-embedding baseline that GR-CLIP is compared against and that it outperforms with far less compute."},{"cited_title":"Zhang, J","cited_arxiv_id":null,"evidence_quote":"Supports the constant-vector approximation of the modality gap used in the method's derivation."},{"cited_title":"Srinivasan, K","cited_arxiv_id":null,"evidence_quote":"Source dataset (Google WIT) for one MixBench subset and for the mean-calibration statistics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source dataset (OVEN) that supplies an existing query-to-multimodal-document retrieval format used in MixBench."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source dataset (VisualNews) for a MixBench subset."}],"review_version":1}