{"id":"5058d76b-ab29-4688-a11e-240c43581640","arxiv_id":"2506.21096","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"By combining CLIP-based cross-modal alignment with ranking distillation from SimCSE and DiffCSE teachers, DALR improves average STS scores by about 0.8 to 1.0 points over KDMCSE across BERT and RoBERTa.","lead":"DALR is a training recipe that makes sentence embeddings learn from images and two text teachers at once, using image-text alignment plus finer text-to-text similarity rankings. On standard semantic similarity and transfer benchmarks it beats the previous multimodal sentence embedding models by about half a point to one point on average.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 11 as printed makes the student text distribution independent of j, so L_IML carries no student gradient unless a subscript typo is corrected and verified in code; the central intra-modal mechanism is therefore undefined as written.","rationale":"The paper's central claim is that the dual-level objective in Eq. 14 improves sentence embeddings; the intra-modal branch is half of that contribution. Eq. 11, on which Eq. 12 depends, is internally inconsistent as printed: it removes the student's dependence from the loss, so the reported w/o-IML ablation cannot be explained by the written objective. This is not a criticism of the empirical style; the results are directionally consistent across STS, transfer, retrieval, and ablation tables, and Appendix E.2's teacher sensitivity analysis is a real strength. However, the method section is the only specification of the architecture, and the GitHub link is not tied to a commit, so a reader cannot tell whether the runnable method uses a corrected numerator. The reader's weakest assumption about teacher pseudo-label reliability is reasonable but less decisive here because Appendix E.2 shows gains persist across SimCSE, DiffCSE, CLIP, and RankCSE teachers; the method appears robust to the specific teacher choice. The more acute risk is that the exact objective of the intra-modal module is undefined as written. I would keep the CONDITIONAL verdict rather than reject, because a one-character fix likely restores the intended distribution-matching loss, but acceptance should require the code check and a corrected equation. A paired significance test against KDMCSE under identical protocols would also strengthen the empirical claim, but I do not make that the primary concern because a corrected Eq. 11 is a precondition for interpreting any of the reported numbers.","tokens_in":19791,"tokens_out":9161,"duration_ms":111807,"concrete_test":"Inspect the released DALR repository and locate the implementation of L_IML / Eq. 12. If the code computes P_{ij}^{t2t} with sim(s_i^z, s_j^{z'}) in the numerator, correct Eq. 11 in the paper and rerun the w/o L_IML ablation to confirm Table 3 reproduces. If the code literally implements the printed Eq. 11, the objective has no student gradient; fix the numerator to index j, retrain, and compare STS/TR against the reported numbers. In either case, release the exact teacher-weighting script so the pseudo-label pipeline is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 11 defines P_{ij}^{t2t} = exp(sim(s_i^z, s_i^{z'})) / sum_{j=1}^N exp(sim(s_i^z, s_j^{z'})). The numerator does not depend on j, and the denominator is the same for every j, so every entry of P_i^{t2t} has the same value; after normalization the student distribution is uniform. Consequently, L_IML in Eq. 12 becomes sum_i KL(Q_i^{t2t} || uniform), a constant with respect to the student parameters that depends only on the frozen teacher entropy and cannot improve or degrade sentence representations. Yet Table 3 reports that removing IML changes STS by about 0.7 points and TR by about 0.5 points. This is an internal inconsistency in the method as written: either the numerator should read exp(sim(s_i^z, s_j^{z'})) (the natural analogue of Eq. 6 and of the P^{t2v} definition), making Eq. 12 a real distribution-matching loss, or the stated objective is degenerate. Because no commit hash or exact code is provided, a reader cannot check which form was actually used. Since IML is one of the two titular contributions in Eq. 14, the reported gains cannot be attributed to intra-modal alignment until this equation is corrected and confirmed against the implementation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DALR, a dual-level alignment framework for multimodal sentence representation learning. The model fine-tunes BERT/RoBERTa students with an image-text InfoNCE loss (Eq. 4), a cross-modal alignment module composed of an auxiliary binary consistency task and KL divergence against teacher vision/text similarity distributions (Eqs. 5-9), and an intra-modal alignment module that combines ListMLE ranking distillation with KL divergence from a weighted SimCSE/DiffCSE teacher distribution (Eqs. 10-13). The joint objective is Eq. (14). Experiments on STS and transfer tasks report that DALR outperforms SimCSE, MCSE, and KDMCSE on BERT and RoBERTa under wiki+flickr and wiki+coco settings; ablations, retrieval evaluations, and MTEB-style downstream tasks are used to support the contribution of each module.","tokens_in":20123,"tokens_out":7702,"duration_ms":80117,"significance":"If the results hold after correction and controlled re-evaluation, DALR is a reasonable incremental contribution: it combines an auxiliary cross-modal consistency task, KL-based distribution matching, and ranking distillation from frozen teachers, and it reports consistent gains over SimCSE, MCSE, and KDMCSE on STS and transfer benchmarks. The paper's strengths include five-seed repetitions for the main STS and ablation tables, evaluation on two multimodal datasets, and ablations that separate the contributions of each loss term. The significance is currently conditional, however, because Eq. (11) as printed makes the intra-modal KL loss degenerate, the training objective omits the loss used for text-only batches, and the baseline comparisons are not controlled under a common seed protocol.","major_comments":[{"comment":"As printed, the numerator exp(sim(s_i^z, s_i^{z'})) and the denominator sum_{j=1}^N exp(sim(s_i^z, s_j^{z'})) in Eq. (11) are both independent of the output index j, so every entry of P_i^{t2t} is identical and the student distribution is uniform. Consequently L_IML in Eq. (12) equals sum_i KL(Q_i^{t2t} || uniform), which is constant with respect to the student parameters. This contradicts Table 3, where removing IML changes the STS average by about 0.7 points and the TR average by about 0.5 points. The numerator should presumably read exp(sim(s_i^z, s_j^{z'})) by analogy with Eq. (6); please correct the equation and confirm against the implementation, since as written the intra-modal alignment loss cannot produce the reported gains. Because no commit hash or exact code is provided, the reader cannot check which form was actually used.","section":"Section 3.3, Eq. (11)"},{"comment":"The final objective in Eq. (14) contains only L_Info, L_CML, and L_IML, but Section 4.1 describes a mixed alternating sampling strategy in which pure-text batches from Wiki1M are also loaded and updated. If those batches are trained with the unsupervised SimCSE objective L_text from Eq. (1), then Eq. (14) omits a term that is active for a large fraction of the training updates; if they are trained with L_Info, the image-text InfoNCE loss is undefined for text-only batches. This omission affects the main results and the ablations (e.g., w/o L_Info in Table 3), so the full training objective must be stated explicitly.","section":"Section 3.4, Eq. (14); Section 4.1"},{"comment":"The STS baselines in Table 1 are taken from the original papers (marks ♡, †, ‡) rather than re-run under the same code, seeds, and evaluation pipeline as DALR, while DALR is averaged over five seeds with its own standard deviations. The reported improvements over KDMCSE are 0.8-0.9 points with DALR standard deviations of 0.5-0.7, so without paired runs or significance tests the claimed superiority is not established. Table 2 reports transfer accuracy without any variance or significance testing, although the gains over KDMCSE are as small as 0.39 points (BERT wiki+flickr). Please rerun all baselines in the same harness with the same seeds and report paired differences or significance tests.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The final text teacher representation is described only as a \"weighted aggregation\" of SimCSE and DiffCSE embeddings, but the weights and any normalization steps are not given. Since the teacher rankings in Eq. (10) and the teacher distribution Q^{t2t} in Eq. (7) are computed from this representation, the missing weights prevent exact reproduction of the pseudo-labels and therefore of the reported results. Please specify the weights and the aggregation procedure, or provide code with a commit hash.","section":"Appendix B, Teacher Text Model"}],"minor_comments":[{"comment":"The text introduces P^{v2t}_{ij} but Eq. (6) defines P^{t2v}_{ij}; please unify the notation for the text-to-image and image-to-text probability names.","section":"Section 3.2, Eq. (6)"},{"comment":"The left-hand side of Eq. (13) is L_IML while the right-hand side uses L_IMA; the subscripts should be consistent.","section":"Section 3.3, Eq. (13)"},{"comment":"There are several typographical errors: \"preform experiments\" should be \"perform experiments\", the reference to Voorhees and Tice has a stray space in \"V oorhees\", and Section 4.2 says \"dedicate the effectiveness\" where \"indicate the effectiveness\" is meant.","section":"Appendix B and references"},{"comment":"The name DALR is misspelled as DLAR in the caption of Figure 3 and in the column headers of Table 10; please correct these.","section":"Figure 3 and Table 10"},{"comment":"The text says \"As illustrated in Figure 9\" but the referenced content is Table 9; please correct the cross-reference.","section":"Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The GitHub link is mentioned but no commit hash is given; I would request code inspection to verify Eq. (11) and the text-only batch loss. The paper also borrows a RankCSE-style teacher setup without clearly acknowledging the source of the weighting strategy, which would be worth a brief note in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the paper's claimed intra-modal alignment module is a no-op as written. Eq. 11 defines the student text distribution with a numerator that doesn't depend on j, so every entry is identical; L_IML in Eq. 12 is a constant KL between the frozen teacher's distribution and a uniform distribution. It cannot affect the student parameters. Yet Table 3 reports that removing IML costs about 0.7 STS points and 0.5 TR points. That's an internal contradiction unless the equation has a subscript typo—almost certainly it should be sim(s_i^z, s_j^{z'})—and the code shows the intended form. As printed, the intra-modal contribution is not supported by the math.\n\nThat said, this isn't a throwaway. The overall design—cross-modal consistency classification plus soft contrastive loss, ranking distillation from multiple text teachers, and KL alignment—is a sensible combination not present in prior work. The empirical coverage is decent: STS, transfer, retrieval, MTEB, ablations, teacher selection, all with five seeds and standard deviations. The authors are honest about borrowing some baseline numbers from earlier papers and about the limitations. The writing is clear.\n\nThe soft spots beyond the equation: no significance tests; several baseline results are lifted from MCSE/KDMCSE papers rather than re-run; Appendix B says the teacher embeddings come from a weighted combination of SimCSE and DiffCSE but never gives the weights; the GitHub link has no commit hash or exact code. The reader's conditional verdict flags some of these, but the stress-test identifies a more concrete problem. The teacher-reliability concern is speculative; the Eq. 11 issue is verifiable from the text.\n\nIf the code confirms the corrected equation, the results are plausible and the paper is a modest but useful increment to the subfield. As presented, it needs a round of revision before it can be trusted.\n\nI'd still send this to peer review. The fix is trivial to demand, and the rest of the paper holds together well enough to warrant referee time. Reviewers should insist on the corrected equation, exact code, and re-run baselines for the central comparison.","headline":"The paper's central intra-modal loss is degenerate as printed, but the overall design is plausible and the equation is likely a fixable typo.","tokens_in":718,"tokens_out":962,"would_cite":false,"duration_ms":53364,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that two alignment losses — a cross-modal consistency loss and an intra-modal ranking-distillation loss — jointly cure the biases that limit multimodal sentence embeddings, improving semantic similarity and transfer…","keywords":["multimodal sentence representation","cross-modal alignment","intra-modal semantic divergence","ranking distillation","semantic textual similarity","contrastive learning","knowledge distillation","KL divergence"],"falsifier":"Run DALR with the text-teacher rankings replaced by random permutations while keeping all other components fixed; if average STS Spearman and transfer accuracy stay within seed noise of the reported 79.5 and 86.92, then ranking distillation is not carrying the claimed weight. A complementary check is to measure DALR's text-text similarity on contradictory pairs such as 'A man is skating' versus 'A man is gliding' and require a separation gain over MCSE and KDMCSE larger than the reported standard deviations.","tokens_in":19611,"feed_emoji":"🔗","tokens_out":6928,"duration_ms":65993,"temperature":0.7,"pith_summary":"Multimodal sentence embeddings learned by contrastive alignment alone inherit two distortions: a caption and its image can be paired even when the caption describes only a minor part of the visual content (cross-modal misalignment bias), and two captions of the same image can be semantically far apart while the pairing treats them as interchangeable (intra-modal semantic divergence). DALR argues that both distortions should be corrected at the loss level, by softening negatives with a consistency task and by replacing binary positive-negative labels with a continuous ranking structure distilled from text teachers. The paper's evidence is a consistent improvement over MCSE and KDMCSE on seven STS datasets and seven transfer tasks with both BERT and RoBERTa, for example DALR-BERT reaches 79.5 average Spearman on wiki+flickr versus KDMCSE's 78.6. If the claim holds, a lightweight unsupervised recipe can improve sentence embeddings without extra data or large-language-model supervision.","feed_headline":"Two-level alignment lifts sentence embeddings to 79.5 Spearman","feed_subtitle":"Adding ranking distillation to cross-modal contrastive learning improves semantic similarity and transfer tasks without extra data.","key_machinery":"The carrying mechanism is the dual-level loss composition in Eq. 14, which stacks three complementary objectives on the same student encoder. The cross-modal part, $L_{\\mathrm{CML}} = L_{\\mathrm{cons}} + L_{\\mathrm{CMA}}$, uses a cosine-embedding consistency loss on shuffled image-text pairs to soften negatives and a KL divergence that pulls the student's normalized text-to-image similarity distribution toward teacher text-text and image-image similarity distributions. The intra-modal part, $L_{\\mathrm{IML}} = L_{\\mathrm{rank}} + L_{\\mathrm{IMA}}$, combines ListMLE, a listwise ranking loss that assigns high probability to the teacher's permutation of similarity scores, with a KL divergence between the teacher text distribution and the student's text distribution under different dropout masks. This machinery converts binary contrastive labels into distributional targets, which is what lets the model carry fine-grained semantic structure across and within modalities.","core_discovery":"The central claim is that the quality ceiling of multimodal sentence representation is set not by the amount of paired image-text data but by two alignment failures: cross-modal misalignment bias and intra-modal semantic divergence. DALR addresses the first by coupling the InfoNCE contrastive objective with an auxiliary binary consistency classifier over shuffled pairs, whose cosine embedding loss produces a soft similarity matrix, and by minimizing the KL divergence between the student's image-to-text distribution and the teacher's text-text and image-image distributions. It addresses the second by treating sentence relationships as a ranking rather than binary positive-negative labels: a weighted ensemble of SimCSE and DiffCSE produces pseudo-ranking labels, the listwise loss ListMLE transfers the ordering, and a KL term aligns the global similarity distribution between teacher and student. The joint objective is $L_{\\mathrm{total}} = L_{\\mathrm{Info}} + \\lambda L_{\\mathrm{CML}} + \\mu L_{\\mathrm{IML}}$ (Eq. 14), trained with mixed alternating batches of Wikipedia text and Flickr or MSCOCO image-text pairs, and on the paper's evidence it produces state-of-the-art STS and transfer scores, with DALR-BERT at 79.5 average Spearman on wiki+flickr and transfer accuracy rising from KDMCSE's 86.53 to 86.92.","pith_inferences":["Beyond the paper, the same two-bias recipe should transfer to other modality pairs such as audio-text or video-text, because image redundancy and caption selectivity are instances of a general dense-modality versus selective-modality asymmetry.","A testable extension is iterative self-distillation: since the teacher-model table shows that stronger text teachers produce better student embeddings, allowing the student's own improved embeddings to replace the frozen text teacher after a few epochs could compound the gains.","The paper specifies only 'weighted aggregation' for the SimCSE and DiffCSE ensemble in Appendix B without giving the weights, so an immediate empirical question is whether a learned or validation-tuned weighting lifts the reported 79.5 average Spearman further; nothing in the method forbids it.","A sharper test of the intra-modal divergence claim than overall STS averages would be human intra-modal similarity judgments on captions that share an image but diverge in focus; DALR should show larger text-text separation for such pairs than MCSE and KDMCSE."],"forward_implications":["Because the two modules are losses rather than data pipelines, they can be dropped onto any contrastive sentence encoder that has a frozen visual teacher; the paper shows that adding the cross-modal alignment module to KDMCSE alone improves Flickr30k retrieval, with image-to-text R@1 rising from 17.9 to 19.1 on wiki+flickr.","Teacher quality directly bounds student quality: Appendix E.2 reports that replacing SimCSE with RankCSE as a text teacher raises DALR's results further, implying that any future improvement in text or image encoders will compound into this method.","The ablation ordering indicates that cross-modal alignment is the primary driver, while intra-modal ranking is a complementary fine-grained regularizer; on wiki+coco, removing $L_{\\mathrm{CML}}$ costs the most in both STS and transfer accuracy.","The gains on reranking, retrieval, classification, and STS shown in Table 5 imply that the learned embeddings transfer beyond the training objective, not just to the two standard benchmarks.","The method's training time is reported as roughly four hours on a single A100 for wiki+flickr with BERT, which is shorter than prior multimodal baselines, so the improvements do not come at a large compute premium."],"supporting_citations":[{"why":"Provides MCSE, the multimodal contrastive baseline that DALR extends, including the InfoNCE setup, the frozen visual/text teacher structure, and the Flickr/MSCOCO evaluation protocol.","marker":"(Zhang et al., 2022a)"},{"why":"Provides KDMCSE, the strongest prior baseline and the false-negative filtering approach that DALR replaces with softened labels and distributional alignment.","marker":"(Nguyen et al., 2024)"},{"why":"Provides SimCSE, the unsupervised text representation baseline and one of the two text teachers that supply pseudo-ranking labels for intra-modal alignment.","marker":"(Gao et al., 2021)"},{"why":"Provides DiffCSE, the second text teacher in the weighted ensemble that produces the ranking and similarity distributions distilled into the student.","marker":"(Chuang et al., 2022)"},{"why":"Supplies the InfoNCE contrastive objective used as $L_{\\mathrm{Info}}$, the base loss around which the dual alignment modules are built.","marker":"(Oord et al., 2018)"},{"why":"Supplies ListMLE, the listwise ranking loss used in $L_{\\mathrm{rank}}$ to transfer the teacher's sentence-ordering structure.","marker":"(Xia et al., 2008)"},{"why":"Supplies CLIP-ViT-B/32, the frozen image teacher whose inter-image similarity distribution drives the cross-modal KL alignment in Eq. 8.","marker":"(Radford et al., 2021)"},{"why":"Supplies the multi-teacher weighted aggregation setup that DALR follows, and an alternative stronger text teacher that further improves results.","marker":"(Liu et al., 2023a)"}],"fun_headline_variants":["Dual-level alignment tackles cross-modal and intra-modal gaps","Ranking distillation refines multimodal sentence embeddings","DALR: soft negatives and ranking distillation align sentences","Fine-grained alignment and ranking ordering boost STS","DALR: dual-level alignment reaches 79.5 Spearman"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen teacher similarities (SimCSE, DiffCSE, and CLIP) are trustworthy enough to act as ranking ground truth, since Appendix B specifies only 'weighted aggregation' for the text teachers without giving the weights, and if those teachers rank divergent captions of one image as near-neighbors, DALR inherits exactly the intra-modal divergence it is meant to remove.","fun_headline_variants_meta":{"raw":{"variants":["Dual-level alignment tackles cross-modal and intra-modal gaps","Ranking distillation refines multimodal sentence embeddings","DALR: soft negatives and ranking distillation align sentences","Fine-grained alignment and ranking ordering boost STS","DALR: dual-level alignment reaches 79.5 Spearman"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3752,"prompt_tokens":962,"completion_tokens":2790,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2712}},"tokens_in":578,"tokens_out":2790,"duration_ms":22028,"temperature":1.0,"reasoning_tokens":2712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:22.048136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DALR with the text-teacher rankings replaced by random permutations while keeping all other components fixed; if average STS Spearman and transfer accuracy stay within seed noise of the reported 79.5 and 86.92, then ranking distillation is not carrying the claimed weight. A complementary check is to measure DALR's text-text similarity on contradictory pairs such as 'A man is skating' versus 'A man is gliding' and require a separation gain over MCSE and KDMCSE larger than the reported standard deviations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides KDMCSE, the strongest prior baseline and the false-negative filtering approach that DALR replaces with softened labels and distributional alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides SimCSE, the unsupervised text representation baseline and one of the two text teachers that supply pseudo-ranking labels for intra-modal alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides DiffCSE, the second text teacher in the weighted ensemble that produces the ranking and similarity distributions distilled into the student."}],"review_version":1}