{"id":"f152c373-864d-4ade-9097-ee5ef3ddfc66","arxiv_id":"2412.10761","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding representation-level and structure-aware distillation from single-modal teachers to cross-modal contrastive training improves both cross-modal and single-modal retrieval under modality imbalance.","lead":"Image-text search systems often assume both images and text carry equal information, but one modality is usually stronger. This paper adds structure-preserving distillation from single-modal teachers so that cross-modal alignment does not damage the stronger modality, improving retrieval on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that cross-modal matching is sub-optimal specifically because of modal imbalance is untested; the method's gains may reflect generic distillation regularization.","rationale":"The reader's weakest_assumption correctly identifies that the direction of imbalance and the causal driver are assumed rather than demonstrated. My concern is more specific: no experiment in the paper manipulates modality sufficiency at all, so the abstract's universal claim about cross-modal matching being sub-optimal 'when imbalanced modalities exist' is not causally tested. The proposed method could be effective simply because it adds strong single-modal supervision and regularization, independent of any imbalance. This is load-bearing because if the mechanism is not imbalance correction, the paper's novelty and framing are substantially weakened, even though the empirical recipe may still work. The paper is otherwise carefully executed: ablations support each loss term, the method is tested across four datasets and three retrieval tasks, and generalization experiments on SCAN, VSRN, and ALBEF add credibility. I therefore do not propose overturning the reader's CONDITIONAL verdict; rather, the controlled test should be a condition for strengthening the paper's central claim.","tokens_in":32443,"tokens_out":6736,"duration_ms":67677,"concrete_test":"Use FLICKR30K with the same X-VLM* pipeline. Systematically vary image sufficiency by adding controlled Gaussian noise or random patch masking at several levels (e.g., 0%, 20%, 40%, 60% corrupted), leaving text unchanged, and conversely vary text sufficiency by word dropout. For each level, train X-VLM* and X-VLM*+ and record I2T/T2I R@1 and single-modal NDCG. If the performance advantage of X-VLM*+ over X-VLM* grows monotonically with the degree of imbalance and reverses when the strong/weak roles are swapped, the rebalancing explanation is supported; if the advantage is constant or appears under balanced conditions, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the Abstract and Section III-A, is that 'ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist.' The support is observational: Table I shows text has higher single-modal retrieval than image, and Figure 1 shows single-modal structure degrades after cross-modal training. Missing is any experiment that manipulates modality sufficiency. The degree and direction of imbalance are never varied, so the correlation between imbalanced single-modal strengths and post-training structural degradation cannot be attributed causally to imbalance. In particular, the proposed method adds representation-level and structure-aware distillation losses (ℓiic, ℓttc, ℓsa) that could improve retrieval through generic regularization, additional supervision, or better teacher features regardless of imbalance. The learned fusion weight λ (≈0.21–0.33 in Section IV-F) gives more weight to the text teacher, which is the already-strong modality, rather than demonstrably 'rebalancing' the weak modality. Without a controlled test, the central claim and the method's 'rebalancing' interpretation are underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses vision-language retrieval under modality imbalance. It claims that exact/ultimate cross-modal matching is generally sub-optimal when the two modalities have different sufficiency, and it proposes a multi-granularity distillation loss composed of representation-level contrastive distillation (ℓiic, ℓttc) and structure-aware distillation (ℓsa) from two single-modal teachers, with a learnable fusion coefficient λ. The student model is X-VLM without the bounding-box module; experiments on MS-COCO (1K and 5K), FLICKR30K, and Vizwiz report cross-modal, single-modal, and mixed retrieval performance, together with ablations, parameter analyses, and generalization experiments on SCAN, VSRN, and ALBEF. The central reported result is that X-VLM*+ improves over X-VLM on I2T and T2I Recall@1 across all four test settings.","tokens_in":32635,"tokens_out":7022,"duration_ms":67376,"significance":"If the empirical results hold, the paper offers a practically useful plug-and-play training regularizer: it consistently improves cross-modal retrieval on four benchmarks and improves or maintains single-modal retrieval relative to most baselines, and the ablation shows that each loss term contributes. The generalization experiments to SCAN, VSRN, and ALBEF strengthen the claim that the module is not tied to a single architecture, and the comparison of MAE/MSE/WD as relational matching criteria is a useful practical detail. The paper does not contain a theoretical derivation or a controlled causal experiment, and the headline causal claim about modality imbalance is not established; the method may be effective for reasons other than 'rebalancing'. The significance is therefore currently that of a strong empirical method rather than a validated mechanism.","major_comments":[{"comment":"The statement that 'ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist' is a universal causal claim, but the support is observational: Table I shows that text encoders perform better than image encoders, Table II shows differences of only 0.1 NDCG points in the S2W/W2S experiments, and Fig. 1 shows structural degradation after cross-modal training. No experiment varies the degree or direction of modality imbalance, so the paper does not demonstrate that imbalance is the causal driver of the observed sub-optimality. Please add controlled experiments that manipulate modality sufficiency (for example, noise-corrupted images, truncated or paraphrased captions, or synthetic settings with reversed sufficiency) or explicitly restrict the claim to the studied text-as-strong/image-as-weak setting.","section":"Abstract and Section III-A"},{"comment":"The structure-aware target is SO = λSI + (1−λ)ST, and Section IV-F reports final learned values of λ = 0.21 on FLICKR30K and λ = 0.33 on Vizwiz. These values put more weight on the text teacher's structure ST, i.e., the 'strong' modality, which is not obviously a rebalancing of the weak image modality; one might instead expect a rebalancing method to emphasize SI. This discrepancy should be explained, and the paper should report ablations with fixed λ = 0 and λ = 1, including single-modal retrieval results, so the reader can see what the learned fusion is actually contributing.","section":"Section III-C and Section IV-F"},{"comment":"The principal empirical claims are reported as single point estimates. Improvements vary substantially across datasets (for example, I2T R@1 gains of 6.2, 11.4, 7.5, and 6.2 in Table III), and some single-modal differences are very small (for example, Table IV, FLICKR30K I2I NDCG@10: 64.1 vs. 64.3). Without variance estimates or significance tests, the reader cannot judge whether the smaller differences are reliable. Please report mean and standard deviation over at least three seeds for the main comparisons and the Table V ablations, or state explicitly that only single runs were performed.","section":"Tables III, IV, and V"}],"minor_comments":[{"comment":"The text says experiments are conducted on 'three datasets' although the paper evaluates on four test settings (MS-COCO 1K, MS-COCO 5K, FLICKR30K, and Vizwiz); please clarify the counting.","section":"Section IV-A"},{"comment":"The abbreviations S2W@Image and W2S@Text are used without expansion; please define them as strong-to-weak and weak-to-strong distillation in the text.","section":"Table II and Section III-A"},{"comment":"The caption introduces 'ViT2BERT' without defining the architecture; please explain what this model is or remove the name if it is an earlier version of the student model.","section":"Fig. 2"},{"comment":"References [9] and [12] appear to be the same paper (Diao et al., AAAI 2021, 'Similarity reasoning and filtration for image-text matching'); please remove the duplicate.","section":"References"},{"comment":"The teacher models are said to use 'unsupervised prototype-aware contrastive learning', but the mechanism is not described in the main text; please provide the objective or point the reader to the specific supplement section with the equations.","section":"Section III-C"},{"comment":"The two cosine similarities d(I,T) and d(T,I) are the same quantity up to argument order; please define the similarity function once and reuse it to avoid notational clutter.","section":"Section III-B, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The empirical method is reasonable and the gains are consistent enough to be worth publishing if the causal narrative is tempered and the robustness of the main numbers is reported. The main risk is the mismatch between the broad claim about modality imbalance and the observational evidence; a controlled manipulation experiment or a clearly restricted claim would address it. I do not see a fundamental correctness error in the proposed loss."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a competent empirical paper whose main value is a plug-and-play training recipe: add two single-modal teacher branches (prototype-aware contrastive) and distill both representation-level and relational structure into a cross-modal student. The gains are consistent and sometimes large on four datasets, including single-modal and mixed retrieval, and the plug-and-play generalization to SCAN, VSRN, and ALBEF is a real plus. The ablation study supports each loss term, with the structure-aware term contributing most. That part deserves credit.\n\nThe soft spot is the interpretive frame. The paper claims that 'ultimate cross-modal matching is generally sub-optimal' under modal imbalance. What they actually show is: (a) text single-modal retrieval is stronger than image retrieval on the four datasets they tried; (b) cross-modal training tends to degrade the stronger modality's structure; (c) their distillation fixes much of the degradation and improves retrieval. That is evidence for the phenomenon in this setup, not a general law. They do have one nice control, Table II, showing that the strong modality can represent the weak but not vice versa, which supports the directionality claim better than the stress-test note gives credit for. But they never vary the degree of imbalance or swap the strong/weak roles, so the causal claim that imbalance is the driver, rather than generic regularization from extra supervision, remains underdetermined. The learnable lambda landing at 0.21–0.33 (more weight to the text teacher) is consistent with their story but does not test it.\n\nOther issues: no error bars or multiple runs, no code release, and the method is largely a combination of known distillation pieces (RKD plus representation contrastive). The novelty is modest but the combination with a learnable fusion weight and the evaluation across three retrieval tasks is a valid contribution.\n\nWho it's for: researchers working on vision-language retrieval or multimodal imbalance who want a practical, well-tested module. It deserves a serious referee; the empirical core is solid enough that a journal or conference should engage with it. The main revision request would be to soften the universal claim and add at least one controlled experiment on imbalance direction or degree.","headline":"Solid plug-and-play distillation recipe for cross-modal retrieval; the imbalance story is plausible but under-tested.","tokens_in":33132,"tokens_out":1570,"would_cite":true,"duration_ms":16370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When text and image modalities are imbalanced, exact instance-level cross-modal matching is sub-optimal; preserving each modality's internal structure while aligning them yields better retrieval.","keywords":["vision-language retrieval","modality imbalance","structure-aware distillation","cross-modal matching","relational knowledge distillation","single-modal retrieval","contrastive learning","multi-granularity distillation"],"falsifier":"Train the same method on a dataset or synthetic modification where images are engineered to be the stronger modality (for example, high-resolution clean images paired with corrupted or repetitive captions). If the method still improves retrieval but the learned fusion weight $\\lambda$ shifts toward the image teacher, the sufficiency-driven explanation is supported; if the structure damage persists regardless of which modality is stronger, or the method fails when the imbalance direction flips, the causal story is wrong.","tokens_in":32228,"feed_emoji":"🖼️","tokens_out":9007,"duration_ms":74639,"temperature":0.7,"pith_summary":"Vision-language retrieval models usually assume the two modalities can represent each other equally, but in practice text is often the stronger modality and image the weaker one. This paper argues that forcing exact instance-level cross-modal matching under that imbalance damages the latent structure of the stronger modality and hurts retrieval, not just single-modal search. To rebalance the learning, the authors add a multi-granularity distillation loss to the standard cross-modal matching loss: two single-modal teacher models transfer their intra-modal similarity structure to the cross-modal student, and the student is trained to keep its relational geometry close to a learned fusion of the two teacher structures. On four benchmarks the augmented model raises image-to-text and text-to-image Recall@1 by up to 11.4 points over its base model and also improves single-modal retrieval. The paper's core claim is that preserving each modality's internal structure while learning cross-modal consistency is the right way to handle imbalanced modalities.","feed_headline":"Structure-aware distillation lifts image-text recall 11.4 points","feed_subtitle":"The method adds single-modal teacher structure to matching, lifting both cross-modal and single-modal retrieval.","key_machinery":"The machinery is a multi-granularity distillation module attached to the cross-modal student model. It consists of representation-level distillation losses ($\\ell_{iic}$, $\\ell_{ttc}$) that align each modality's [CLS] embeddings with the corresponding single-modal teacher, and a structure-aware distillation loss ($\\ell_{sa}$) computed as the MAE between the student's cross-modal relational matrix $S_{IT}$ and a fused teacher matrix $S_O = \\lambda S_I + (1-\\lambda) S_T$, where $\\lambda$ is learned. The teachers are trained with prototype-aware contrastive learning so that semantically similar instances are not treated as negatives. The total objective is $\\mathcal{L} = \\ell_{cr} + \\ell_{md}$, with $\\ell_{cr} = \\ell_{itc} + \\ell_{itm}$.","core_discovery":"The paper's central claim is that 'ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist.' Using single-modal retrieval performance as a proxy for modality sufficiency, the authors identify text as the strong modality and images as the weak one across MS-COCO, FLICKR30K, and Vizwiz. They show that standard cross-modal training degrades the text model's text-to-text retrieval more than it improves the image model's image-to-image retrieval, and that distillation from the strong to the weak modality helps the weak modality while distillation from the weak to the strong hurts the strong one. Their proposed solution, X-VLM*+, keeps the cross-modal matching losses (ITC and ITM) but adds representation-level distillation and structure-aware distillation, so the learned common space preserves the intra-modal structure of both single-modal teachers. The experiments report consistent gains in cross-modal, single-modal, and mixed retrieval across four datasets, including Recall@1 gains of 6.2/11.4/7.5/6.2 for image-to-text and 3.1/5.1/7.9/6.5 for text-to-image over X-VLM.","pith_inferences":["A direct test of the paper's causal story is to reverse the imbalance (for example, pair unusually clean, detailed images with noisy or repetitive captions) and check whether the learned $\\lambda$ shifts toward the image teacher; the sufficiency-based diagnosis predicts it would.","The relational-consistency term acts as a structure regularizer on top of contrastive alignment, so the same multi-teacher distillation design could transfer to other imbalanced multi-modal retrieval settings, such as audio-text or video-text search.","Because the module is architecture-agnostic, it could be appended when fine-tuning large pre-trained vision-language models to prevent single-modal representation collapse, a use the paper demonstrates on X-VLM but does not claim as a general recipe."],"forward_implications":["Treating instance-level matching as the only objective is suboptimal when modalities are imbalanced; adding structure preservation raises cross-modal recall.","The extra distillation works as a plug-and-play module: adding it to SCAN, VSRN, ALBEF, and X-VLM improves their retrieval on the tested datasets.","Single-modal retrieval also improves, meaning the method partially reverses the degradation of the strong modality that cross-modal training usually causes.","The learned fusion coefficient $\\lambda$ settles near 0.21 and 0.33, indicating that the text (strong) teacher contributes more structural guidance than the image (weak) teacher.","Applying the module during fine-tuning of a large pre-trained model improves single-modal retrieval while keeping cross-modal performance, so the idea is not limited to training from scratch."],"supporting_citations":[{"why":"Supplies the proposition that modality sufficiency tracks optimal single-modal performance, the criterion used to declare text the strong modality.","marker":"[16]"},{"why":"Provides the strong/weak modality competition result that joint training hurts the stronger modality, the mechanism the paper tests in its imbalance exploration.","marker":"[19]"},{"why":"X-VLM is the cross-modal student model whose ITC and ITM losses form the base objective.","marker":"[35]"},{"why":"Relational knowledge distillation is the source of the matrix-level geometric consistency used in structure-aware distillation.","marker":"[39]"},{"why":"Prototype-aware contrastive learning trains the single-modal teachers without treating semantically similar instances as negatives.","marker":"[40]"},{"why":"SCAN, used in the exploratory experiment that shows cross-modal training degrades text-to-text retrieval.","marker":"[6]"},{"why":"BLIP, used in the same exploratory experiment to demonstrate strong-modality degradation.","marker":"[14]"},{"why":"CYCLIP is the main structural-regularization baseline, showing the proposed distillation strategy outperforms multi-task structural regularization.","marker":"[51]"},{"why":"UMT is the uni-modal-teacher distillation baseline whose instance-relation distillation approach the paper compares and extends.","marker":"[52]"}],"fun_headline_variants":["Structure-aware distillation rebalances vision-language retrieval","Rebalanced matching: structure-aware distillation boosts recall","Preserving intra-modal structure improves cross-modal retrieval","Modality imbalance? Structure-aware distillation is the fix","Distilling structure rebalances multimodal matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the measured single-modal retrieval gap between text and image reflects a real difference in modality sufficiency, and that this asymmetry, rather than optimizer dynamics or architecture capacity, is what causes cross-modal training to damage single-modal structure.","fun_headline_variants_meta":{"raw":{"variants":["Structure-aware distillation rebalances vision-language retrieval","Rebalanced matching: structure-aware distillation boosts recall","Preserving intra-modal structure improves cross-modal retrieval","Modality imbalance? Structure-aware distillation is the fix","Distilling structure rebalances multimodal matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000421,"raw_usage":{"total_tokens":2206,"prompt_tokens":1027,"completion_tokens":1179,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1109}},"tokens_in":643,"tokens_out":1179,"duration_ms":9608,"temperature":1.0,"reasoning_tokens":1109,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:37:44.172367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same method on a dataset or synthetic modification where images are engineered to be the stronger modality (for example, high-resolution clean images paired with corrupted or repetitive captions). If the method still improves retrieval but the learned fusion weight $\\lambda$ shifts toward the image teacher, the sufficiency-driven explanation is supported; if the structure damage persists regardless of which modality is stronger, or the method fails when the imbalance direction flips, the causal story is wrong.","supporting_citations":[{"cited_title":"Co-training with insufficient views,","cited_arxiv_id":null,"evidence_quote":"Supplies the proposition that modality sufficiency tracks optimal single-modal performance, the criterion used to declare text the strong modality."},{"cited_title":"Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably),","cited_arxiv_id":null,"evidence_quote":"Provides the strong/weak modality competition result that joint training hurts the stronger modality, the mechanism the paper tests in its imbalance exploration."},{"cited_title":"Multi-grained vision lan- guage pre-training: Aligning texts with visual concepts,","cited_arxiv_id":null,"evidence_quote":"X-VLM is the cross-modal student model whose ITC and ITM losses form the base objective."},{"cited_title":"Relational knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"Relational knowledge distillation is the source of the matrix-level geometric consistency used in structure-aware distillation."},{"cited_title":"Large-margin contrastive learning with distance polarization regularizer,","cited_arxiv_id":null,"evidence_quote":"Prototype-aware contrastive learning trains the single-modal teachers without treating semantically similar instances as negatives."},{"cited_title":"Stacked cross attention for image-text matching,","cited_arxiv_id":null,"evidence_quote":"SCAN, used in the exploratory experiment that shows cross-modal training degrades text-to-text retrieval."},{"cited_title":"BLIP: boot- strapping language-image pre-training for unified vision- language understanding and generation,","cited_arxiv_id":null,"evidence_quote":"BLIP, used in the same exploratory experiment to demonstrate strong-modality degradation."},{"cited_title":"Cyclip: Cyclic contrastive language-image pretraining,","cited_arxiv_id":null,"evidence_quote":"CYCLIP is the main structural-regularization baseline, showing the proposed distillation strategy outperforms multi-task structural regularization."}],"review_version":1}