{"id":"be1a9838-0724-4663-b803-63e17ca850d2","arxiv_id":"2412.14384","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Standardizing per-modality embeddings can shrink the CLIP modality gap to near zero, but the post-hoc version's evaluation is circular because it uses test-set statistics.","lead":"This paper proposes two ways to shrink the gap between CLIP's image and text embeddings: subtract each modality's average embedding and re-normalize, or add separate batch-normalization layers during training. The authors claim this produces a better image-caption scoring metric, but the main post-hoc result is weakened because it uses the test data itself to compute the averages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"I0Tpost's near-zero modality gap is an artifact of estimating the standardization mean vectors on the same test set used for evaluation; no held-out evidence supports the method's generality as a zero-gap procedure or as a metric.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: I0Tpost's mean vectors are computed from the same test set on which the modality gap and downstream metrics are measured. This is not a style issue or a mere missing baseline; it directly undermines the headline result. Once the mean vectors are allowed to come from the evaluation set, the near-zero centroid distance is a mathematical consequence of the standardization step, and the reported retrieval and correlation numbers for I0Tpost are evaluated under a protocol that no deployed system could use. The paper discloses the reliance but does not test the method's out-of-sample behavior, so the central claim remains unsupported. I0Tasync is more defensible but achieves only a moderate gap, and the paper does not separately establish the 'towards zero' claim with that variant. Because the primary contribution is invalidated by the evaluation design, the existing REJECT verdict remains appropriate. I would not change the verdict. The proposed check is a minimal, decisive way to determine whether the concern is fatal or merely a missing experiment: if the near-zero gap and retrieval benefits survive on a held-out split, the method would have real value, but the present paper does not provide that evidence.","tokens_in":16214,"tokens_out":3832,"duration_ms":34743,"concrete_test":"On Flickr30k, construct disjoint estimation and evaluation splits (for example, the Karpathy training split versus the test split). Estimate the per-modality mean vectors bar_x and bar_y using only the estimation split, then apply x' = Normalize(x - bar_x) and y' = Normalize(y - bar_y) to the evaluation split. Report CD, LS, I2T/T2I R@1, and Flickr8k-Expert correlation using only these held-out statistics. If CD remains below the paper's 0.19 low-severity threshold and retrieval stays within about one point of Table 2, the circularity concern is resolved; if CD rises substantially or retrieval degrades, the near-zero gap is a same-set artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that I0Tpost reduces the modality gap to approximately zero (CD 0.0102, LS 0.5374 on Flickr30k) while preserving or improving retrieval, and that it can serve as a general post-hoc replacement for CLIPScore. This claim depends on the assumption that the per-modality mean vectors used in standardization are stable, transferable properties of the deployment distribution. In the paper, however, those mean vectors are estimated from the same Flickr30k test split whose centroid distance, linear separability, retrieval, and correlation are then measured. Subtracting the empirical mean of a set and re-normalizing forces the new centroid difference to be small by construction, so a CD of 0.0102 is not evidence that a modality gap has been removed; it is largely a consequence of using the same data to estimate the statistics and to evaluate the result. The same logic explains the drop in linear separability. The paper acknowledges this in Section 8 ('I0Tpost relies on the entire test dataset'), but it does not provide any disjoint-split experiment showing that the near-zero gap persists when the mean vectors are estimated from one set and applied to another. Without such an experiment, the method's usefulness as an automatic evaluation metric on unseen captions, and even its status as a general post-hoc procedure, is unverified. The trainable I0Tasync variant avoids the circularity but only reaches CD 0.4795, a moderate gap, so it cannot support the headline claim of a near-zero modality gap. Thus the strongest empirical result in the paper is an artifact of the evaluation protocol, and the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes I0T, a two-stage framework to reduce the modality gap in CLIP embeddings. Stage one is an optional plug-in fine-tuning strategy based on Long-CLIP with cyclic losses and MCSIE; stage two consists of two alternatives: I0Tpost, which standardizes normalized image and text embeddings by subtracting per-modality mean vectors and re-normalizing, and I0Tasync, which adds separate batch-normalization layers trained asynchronously on frozen encoders. The authors report that I0Tpost reduces centroid distance to 0.0102 and linear separability to 0.5374 on Flickr30k while preserving or improving retrieval, and that I0Tasync reaches a moderate gap of 0.4795. They further propose I0T-S as an explainable reference-free image-captioning metric based on I0Tpost.","tokens_in":16508,"tokens_out":8346,"duration_ms":74861,"significance":"If the claims were established, the paper would make a useful practical contribution: a simple post-hoc transformation that removes a large part of the modality gap without retraining, plus a trainable alternative, with code released and a broad set of downstream evaluations. The connection to CLIPScore is also interesting and could lead to more interpretable evaluation scores. The paper deserves credit for releasing code, for checking both centroid distance and linear separability, and for including ablations on batch size and normalization choices. However, the central evidence for I0Tpost is currently not independent: the same test set supplies the standardization statistics and the evaluation numbers, so the near-zero centroid distance is largely a consequence of the evaluation protocol. The paper is transparent about this in its Limitations section, but transparency does not make the result supportive.","major_comments":[{"comment":"The headline result for I0Tpost is circular. The per-modality mean vectors xbar and ybar are computed from the same Flickr30k test split on which centroid distance, linear separability, retrieval, and correlation are then measured. Since the centroid distance is defined as the distance between the modality means, subtracting the empirical mean of the same set and re-normalizing removes the first-order gap by construction; the reported CD of 0.0102 is therefore not independent evidence that a modality gap has been removed. The same reasoning explains the drop in linear separability. The paper acknowledges in Section 8 that I0Tpost relies on the entire test dataset, but it does not provide the necessary control experiment: estimate xbar and ybar on a disjoint training or validation split, apply them to the test split, and report CD, LS, retrieval, and metric correlation. Without such an experiment, the claim that I0Tpost reduces the modality gap to approximately zero, and the related claim that I0T-S is a usable evaluation metric on unseen captions, are unsupported. I0Tasync avoids this circularity but reaches only CD 0.4795, a moderate gap, so the paper's central 'zero gap' claim rests entirely on the circular I0Tpost result.","section":"Section 4.2, Section 5.2, Table 2, Section 8"},{"comment":"The comparison with PAC-S and BLIP is confounded by different checkpoints. According to Appendix D.3, PAC-S and BLIP are evaluated using the provided checkpoints, while the I0T models are Long-CLIP-based models fine-tuned on COCO with the authors' first-stage recipe. Differences in retrieval and correlation could therefore come from the base model and training data rather than from the proposed method. A fair comparison should apply I0Tpost to the same base CLIP checkpoint and train or fine-tune all comparison methods under an identical protocol, or at least include a matching-control experiment. This is load-bearing because the paper claims a 9.2% improvement in T2I retrieval and competitiveness with PAC-S based on these tables.","section":"Appendix D.3, Table 2"},{"comment":"The main tables and figures report point estimates without error bars or significance tests. Many of the claimed differences are small relative to the likely run-to-run variability (for example, the relative correlation values on Expert and CF in Table 2 differ by only a few points across models). The paper states in Appendix D.1 that three random seeds were used for dataset selection, but the main results do not report mean and standard deviation over seeds. Without this, the reader cannot assess whether the reported improvements and the ranking in Table 2 are statistically reliable. Please report mean +/- std over at least three seeds for all main metrics, or clearly state if only a single run is reported.","section":"Tables 1, 2, 4, 6, 7 and Figure 8"}],"minor_comments":[{"comment":"The upper-bound proof assumes that all non-peak activations are uniformly distributed, an assumption that is stated but not justified empirically or theoretically. The derived limit is therefore not a general upper bound for the cosine similarity of arbitrary normalized embeddings with peak activations; the authors should clarify the scope of the claim or support the uniform assumption with activation statistics.","section":"Section 3.3 and Appendix C"},{"comment":"The linear separability values for MG and CLIPpost in Table 7 are negative (-1.4094 and -1.8497), which is inconsistent with the definition LS = 1 - MSE if the model is trained and evaluated as described. The authors should explain how negative values arise or fix the computation.","section":"Table 7"},{"comment":"The claim that I0T-S is more explainable than CLIP-S is supported mainly by anecdotal examples in Figures 1 and 5. It would be stronger to report the full distribution statistics (mean, variance, range) of the cosine similarity scores over the test set, as well as a quantitative comparison of the ordering of correct versus incorrect pairs.","section":"Section 5.3 and Figure 5"},{"comment":"The term 'reference-free' for I0T-S is misleading in the single-sample setting: to compute the text mean vector for standardization, the method needs a set of captions from the test distribution, so it is not a purely single-sample metric. The authors should qualify this terminology and state the minimum corpus size required for stable estimates.","section":"Section 8"},{"comment":"The retrieval protocol in Table 4 differs from the common practice of using all five captions for Flickr30k. The paper should clearly define the one-caption versus five-caption evaluation and report both consistently, since the choice affects the I2T/T2I balance and the comparability with prior work.","section":"Table 4 and Appendix D.2"},{"comment":"There is a typo in the Ethical Statement: 'ClIP-S' should be 'CLIP-S'.","section":"Ethical Statement"}],"recommendation":"major_revision","confidential_remarks":"The reader's report correctly identifies the circularity of the I0Tpost evaluation as the central problem. I recommend major revision rather than outright rejection because the flaw is in the evaluation protocol and can be addressed by a holdout experiment in which the standardization statistics are estimated on a disjoint split and applied to the test set. If that experiment fails to show a near-zero centroid distance with train-estimated means, the paper's main claim would collapse and the manuscript should then be rejected. I also strongly recommend the editor require the checkpoint-fairness comparison and error bars as conditions for the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper has a genuinely interesting observation about peak activations and a sensible trainable BN variant, but its flagship post-hoc result (near-zero modality gap) is largely an artifact of the evaluation protocol. The paper's own Limitations section admits I0Tpost relies on the entire test dataset.\n\nWhat's actually new: the analysis attributing the gap to modality-specific mean vectors rather than just a few peak dimensions is a real addition. The clipping experiment (clipping peaks doesn't help) is a good negative result. I0Tasync—separate batch-norm layers trained asynchronously on frozen encoders—is a reasonable, modest contribution, and the paper is honest that it only reaches a moderate gap. The I0TScore idea is a natural extension and is evaluated with human correlation, which is a useful sanity check.\n\nSoft spots: the main table's I0Tpost row is built by subtracting the per-modality mean computed from the same Flickr30k split used for retrieval and gap measurement. Centroid distance near zero is then by construction. The paper's Section 8 admits this, but the abstract still claims 'reduces the modality gap approximately to zero' as if it were a model property. There is no held-out experiment showing the means transfer to a different set, which would be needed to support the metric claim. The comparison with PAC-S and BLIP uses their released checkpoints, so the retrieval improvements over them are not apples-to-apples. Main tables lack error bars, though the appendix shows some stability across seeds and batch sizes. The async-BN variant is less flashy but is the more defensible piece; it just doesn't live up to the title.\n\nWho it's for: researchers working on CLIP evaluation metrics or modality gap. It deserves a serious referee, but the authors should be pushed to add a disjoint-split experiment and tone down the post-hoc claims. I'd treat the post-hoc CD as a property of the standardization trick, not of the model.\n\nRecommendation: send it to review with a clear request for held-out evaluation and revised claims.","headline":"Useful analysis of CLIP's modality gap, but the headline post-hoc result is an artifact of evaluating on the same set used to estimate the standardization statistics.","tokens_in":17084,"tokens_out":1668,"would_cite":false,"duration_ms":14609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By subtracting each modality's mean embedding and re-normalizing, this paper nearly eliminates CLIP's modality gap while preserving, and often improving, downstream performance.","keywords":["modality gap","CLIP","embedding standardization","image-text retrieval","batch normalization","reference-free evaluation","centroid distance","cosine similarity"],"falsifier":"Compute I0Tpost means on the Flickr30k training split, apply them to the Flickr30k test split, and report centroid distance on test: if CD jumps well above 0.19, the near-zero gap is an artifact of test-set statistics; alternatively, evaluate on COCO captions with means computed on Flickr30k.","tokens_in":15962,"feed_emoji":"📐","tokens_out":6258,"duration_ms":49006,"temperature":0.7,"pith_summary":"Contrastive language-image models such as CLIP place images and texts in separate regions of embedding space, a 'modality gap' that breaks the intended semantic comparison. The paper claims this gap is caused by modality-specific activation patterns, especially peak activations on a few fixed dimensions, that each encoder learns, and that removing those patterns brings the gap to near zero. Its post-hoc method, I0Tpost, subtracts the per-modality mean vector from normalized embeddings and re-normalizes, cutting centroid distance to 0.0102 and linear separability to 0.5374 on Flickr30k. The paper also proposes a trainable version, I0Tasync, that adds per-modality batch normalization layers and reaches a moderate gap without test-set access. A reader should care because a nearly closed gap makes CLIP usable as an explainable, reference-free image-text evaluation metric and clarifies what the embeddings actually encode.","feed_headline":"Subtracting per-modality means nearly eliminates CLIP's modality gap","feed_subtitle":"Standardized CLIP embeddings cut centroid distance to 0.0102 on Flickr30k and improve retrieval while staying explainable.","key_machinery":"The central object is the 'modality-specific characteristic' in normalized embeddings: mean activation vectors that are nearly constant across samples of one modality, with peaks in fixed dimensions (for Long-CLIP, a negative peak at dimension 93 for images and positive peaks at 134 and 313 for texts). The argument-carrying mechanism is standardization of each modality's normalized embeddings by subtracting its own mean vector and renormalizing with the Frobenius norm, x'_i = Normalize(x_i - \\bar{x}), y'_i = Normalize(y_i - \\bar{y}). For the trainable variant the same statistics are learned by independent batch normalization layers placed after each encoder and trained asynchronously with the multimodal MCSIE objective. A supporting analytic claim shows that peak activations p and q cap the attainable cosine similarity at $\\sqrt$((1-$p^{2}$)(1-$2q^{2}$)), so removing peaks and mean structure is what permits similarity scores to approach their semantic range.","core_discovery":"On the paper's account, CLIP's image and text encoders do not merely differ by where their data live; each encoder stamps every embedding with a characteristic activation profile, negative peaks in the same dimensions for all images and positive peaks in other dimensions for all texts, that limits how close cross-modal cosine similarity can get. The discovery is that discarding this modality-specific signature across all dimensions, not just clipping the peaks, collapses the gap. The post-hoc procedure subtracts the mean of all normalized image embeddings from each image embedding and the mean of all normalized text embeddings from each text embedding, then re-normalizes by Frobenius norm; on frozen encoders this yields centroid distance 0.0102 from 0.7642 and linear separability 0.5374 from 0.9985 on Flickr30k, while text-to-image retrieval improves to 76.30. The same idea is made trainable as I0Tasync with separate batch normalization layers per modality, reducing the gap to a moderate level without needing the evaluation distribution. The paper presents I0Tpost, under the name I0T-S, as an automatic reference-free captioning metric whose raw cosine scores are spread across a wider, interpretable range than CLIPScore.","pith_inferences":["If the per-modality mean vectors are stable, I0Tpost implies CLIP embeddings are approximately mean-shift separable; one testable extension is calibrating the means on a small unlabeled in-domain pool at deployment time and checking whether near-zero CD persists across datasets.","The peak-activation diagnosis is a cheap diagnostic: detecting the same peak dimensions across samples of one modality could flag modality-specific artifacts in other contrastive models, not only CLIP.","I0T-S's interpretable absolute scores suggest reference-free evaluation metrics could be standardized in place of scaling cosine similarities, a change that would make captioning scores comparable across domains; this is an extension beyond the paper's claims.","The trainable variant leaves a moderate gap, so a natural next step the paper does not take is combining async BN statistics with I0Tpost-style standardization at inference to get near-zero gap without test-set means."],"forward_implications":["I0Tpost brings Flickr30k centroid distance from 0.7642 to 0.0102 and linear separability from 0.9985 to 0.5374, with text-to-image retrieval rising from 67.10 to 76.30.","I0Tasync reaches a moderate gap (CD 0.4795) without access to the test distribution, adding about 10M parameters, and improves T2I retrieval over the starting CLIP.","Using I0Tpost as I0T-S, cosine similarity between an image and its correct caption is higher and, for a wrong caption, lower than CLIP-S, without the 2.5 scaling CLIPScore needs.","The paper explicitly does not claim the gap and downstream performance are causally linked; the result is that the gap can be cut without sacrificing retrieval or classification.","The same standardization idea works on a ResNet-based CLIP (CLIPpost), suggesting the mechanism is not tied to ViT architectures."],"supporting_citations":[{"why":"Supplies CLIP, the contrastive image-text model whose embeddings are diagnosed and standardized.","marker":"Radford et al., 2021"},{"why":"Defines the modality gap as centroid distance and provides Mind-the-Gap, the post-hoc shifting baseline I0Tpost is compared against.","marker":"Liang et al., 2022"},{"why":"Provides the linear separability metric and the framing of the contrastive gap that the paper adopts for quantifying severity.","marker":"Fahim et al., 2024"},{"why":"Supplies CLIPScore, the reference-free evaluation metric that I0T-S is proposed to replace.","marker":"Hessel et al., 2021"},{"why":"Supplies PAC-S, the state-of-the-art contrastive-based captioning metric used as a downstream baseline.","marker":"Sarto et al., 2023"},{"why":"Provides the CyCLIP cyclic losses used in the first-stage fine-tuning objective LCyCLIP.","marker":"Goel et al., 2022"},{"why":"Provides Long-CLIP, the backbone the I0T framework builds on for first-stage semantic enhancement.","marker":"Zhang et al., 2024"},{"why":"Provides MCSE, the multimodal contrastive sentence embedding method re-implemented as MCSIE for training the BN layers.","marker":"Zhang et al., 2022a"},{"why":"Supplies Unif-Align, a baseline that combines alignment and uniformity objectives for comparison.","marker":"Wang and Isola, 2020"}],"fun_headline_variants":["Per-modality centering shrinks CLIP's gap to near zero","Subtract per-type means to collapse CLIP's modality gap","Standardizing CLIP embeddings kills the modality gap","I0Tpost: zeroing CLIP's gap with per-encoder mean removal","Frozen CLIP, no gap: I0T post-hoc standardization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The near-zero centroid distance is measured with mean vectors computed from the same test set whose embeddings are standardized, so the method assumes those per-modality means are stable properties of the deployment distribution rather than a quirk of that particular test split.","fun_headline_variants_meta":{"raw":{"variants":["Per-modality centering shrinks CLIP's gap to near zero","Subtract per-type means to collapse CLIP's modality gap","Standardizing CLIP embeddings kills the modality gap","I0Tpost: zeroing CLIP's gap with per-encoder mean removal","Frozen CLIP, no gap: I0T post-hoc standardization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1522,"prompt_tokens":993,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":609,"tokens_out":529,"duration_ms":5635,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:16:33.788824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute I0Tpost means on the Flickr30k training split, apply them to the Flickr30k test split, and report centroid distance on test: if CD jumps well above 0.19, the near-zero gap is an artifact of test-set statistics; alternatively, evaluate on COCO captions with means computed on Flickr30k.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the modality gap as centroid distance and provides Mind-the-Gap, the post-hoc shifting baseline I0Tpost is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CyCLIP cyclic losses used in the first-stage fine-tuning objective LCyCLIP."}],"review_version":1}