REVIEW 5 major objections 6 minor 2 cited by
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
T0 review · 5 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Unpaired images and text can replace most of the paired supervision needed to align vision and language encoders.
desk verdict A solid two-stage alignment method whose gains hold when unpaired data match the paired distribution, but the abstract overclaims cross-dataset robustness and the evidence lacks error bars and a random-teacher control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the KLOT divergence between entropic optimal transport plans, KL(OT_{ε*}(K*) || OT_ε(K)), which compares the entire matching structure between a batch of images and texts rather than only their nearest neighbors. OT_ε(K) is the entropic optimal transport plan of the affinity matrix K, a soft permutation that encodes relational geometry. The load-bearing result is Theorem 5.1: the gradient of KLOT with respect to K equals (OT_ε(K) − OT_{ε*}(K*))/ε*, a closed-form expression that avoids unrolling or implicit differentiation of Sinkhorn. This makes it possible to train with batches of 32k samples, which is what lets the method exploit up to 1M unpaired samples.
What would settle it
Take a pair of encoders with mutual k-NN agreement below the range plotted in Figure 9 (about 0.14), or paired data from one domain and unpaired data from a very different domain, and measure the linear teacher's top-1 cross-modal retrieval on a held-out subset of the unpaired distribution; if the teacher performs at chance, K* is uninformative and SOTAlign should collapse to the supervised baseline. Alternatively, train SOTAlign with unpaired data whose spherical sliced Wasserstein distance from the paired set exceeds the range mapped in Figure 5, and test whether the gain over the supervised
Extended reading notes
Core claim
SOTAlign aligns unimodal encoders in two stages. Stage one fits a linear alignment model (CCA, Procrustes, or contrastive) on the small paired set, recovering a coarse shared geometry that already yields strong zero-shot retrieval. Stage two uses this linear model as a teacher: on each batch of unpaired images and texts, its affinity matrix K* serves as a target geometry, and the alignment layers are trained with a supervised loss on pairs plus a regularizer that keeps the learned geometry close to K*. The regularizer is the KLOT divergence, the KL divergence between entropic optimal transport plans KL(OT_{ε*}(K*) || OT_ε(K)). The main theoretical result is an explicit gradient, ∇_K KLOT = (
Load-bearing premise
The load-bearing premise is that the linear map fit on a few thousand paired samples generalizes to arbitrary unpaired data, so that the teacher's predicted affinity on unpaired batches is semantically meaningful; if the relationship between the frozen encoders stops being approximately linear outside the paired distribution, the KLOT regularizer will amplify the teacher's errors.
Editorial extensions
If this is right
- If the linear teacher captures the cross-modal geometry of frozen encoders, then unpaired images and texts can refine alignment without any additional paired supervision; the paper shows consistent gains up to 1M unpaired samples, with diminishing returns after about 500k.
- The closed-form gradient for KLOT applies to any OT-based loss of the form KL(T || OT(K)), removing a scalability barrier for OT-based representation learning; the paper explicitly notes this generality for recent OT-based alignment and contrastive methods.
- The method is robust to distribution shift: mixing unpaired images from one corpus and unpaired text from another still improves COCO and ImageNet performance, and the paper's spherical sliced Wasserstein distance predicts the size of the gain before training.
- The paired-data requirement drops by roughly a factor of four: 10k pairs plus unpaired data match a supervised model trained on about 40k pairs, and the largest gains appear in the intermediate regime between 1k and 10k pairs.
Reading between the lines
- A natural untested extension is iterative self-training: use the refined alignment as the new teacher and repeat, which would test whether the linearity of the initial teacher is a hard ceiling or just a starting point.
- If the underlying hypothesis about convergent representation geometry holds beyond vision and language, the same two-stage recipe should work for other modality pairs with large unpaired corpora, such as audio-text or molecule-text; the paper only validates vision-language.
- The observed link between sliced Wasserstein distance and gain suggests a practical selection rule: measure candidate unpaired corpora before training and only add data below a distance threshold, a heuristic the paper does not state explicitly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SOTAlign, a two-stage semi-supervised method for aligning frozen unimodal vision and language encoders. In stage one, a linear alignment map (Procrustes, CCA, or contrastive) is fit on a small set of paired image–text samples. In stage two, lightweight alignment layers f and g are trained with a supervised SigLIP loss on the same pairs plus an optimal-transport-based regularization (KLOT) that encourages the affinity structure of unpaired batches to match the structure induced by the linear teacher. The authors report consistent gains over supervised and semi-supervised baselines in zero-shot retrieval and classification when unpaired data are drawn from the same distribution as the paired data (e.g., CC3M), and claim robustness across datasets and encoder pairs. They also derive an explicit gradient expression for KLOT, removing the need to backpropagate through Sinkhorn iterations.
Significance. If the claims are sustained, the paper makes a useful contribution: it demonstrates that in-distribution unpaired unimodal data can substitute for a large fraction of paired supervision when aligning frozen encoders, and it provides a scalable OT-based regularizer with an explicit gradient. The code release and the careful handling of the Sinkhorn backpropagation memory bottleneck are strengths. However, the central empirical claim is currently broader than the evidence: the method's own cross-domain results show that the linear teacher's target geometry degrades under distribution shift, and the absence of error bars and of a control for the teacher's semantic content prevents the reader from attributing the gains to the proposed mechanism.
major comments (5)
- [Theorem 5.1 / Appendix C.2] The main-text gradient formula contradicts the appendix. Eq. (13) states ∇_K KLOT(K||K*) = (OT_ε(K) − OT_{ε*}(K*))/ε*, but KLOT is defined as KL(OT_{ε*}(K*) || OT_ε(K)). With T = OT_{ε*}(K*) fixed, Theorem C.4 (Eq. 28) gives ∇_K KL(T || OT_ε(K)) = (OT_ε(K) − T)/ε, i.e., the denominator should be ε, not ε*. Since the explicit gradient is a central claim and is used in Algorithm 1 to update the alignment layers, this discrepancy must be corrected and the implementation checked against the correct formula.
- [Table 7 / Section 6.2] The text states that SOTAlign 'consistently outperforms the supervised baseline' across heterogeneous unpaired sources, but Table 7 contradicts this. With CC3M pairs and ImageNet images + WikiText103 unpaired text, COCO T2I R@1 is 20.7 versus 21.0 for supervised SAIL, and I2T is 28.1 versus 27.4; with COCO images + WikiText103, T2I drops to 19.5. Thus the method does not leverage unpaired data under a substantial distribution shift. The abstract's claim of 'robust joint embeddings across datasets' and the 'consistently outperforms' wording in Section 6.2 need to be qualified to the in-distribution or mild-shift regime.
- [Algorithm 1 / Eq. (4)-(5)] The mechanism by which unpaired data improve performance is not isolated. The loss always contains the supervised SigLIP term on the same 10k pairs, and the KLOT term adds an extra training signal on unpaired batches. Without an ablation that replaces the teacher target K* with a random, shuffled, or identity affinity matrix on unpaired batches, the gains could in principle come from the OT regularizer's smoothing effect or from the linear teacher's initialization rather than from the semantic validity of K*. The manuscript should report such a control. Additionally, Algorithm 1 line 2 ('Initialize encoders f and g') does not specify whether the linear projections Wx, Wy are used as initialization; if they are, the comparison to SAIL (which starts from random init) is confounded.
- [Tables 1, 4, 5, 7] All main results are single numbers with no standard deviations or number of seeds. In the 10k-pair regime, random batch composition and optimization variability can easily account for 1–2 R@1 points, and several claimed gains are in that range (e.g., SOTAlign with ImageNet+WikiText vs SAIL, 20.7 vs 21.0). The paper should report mean ± std over at least three seeds for the central comparisons, and ideally for the hyperparameter-selected configuration.
- [Figure 9 / Appendix B.2] The support for the linear teacher's transferability is weak. Pearson r = 0.83 is computed over only 6 encoder pairs, and the text itself concedes that 'additional data points will be required to draw firm conclusions.' This does not establish that the linear map produces a semantically meaningful target K* on arbitrary unpaired batches; indeed, Table 7 shows it does not under distribution shift. The paper should either provide direct evidence of K* quality on held-out unpaired data (e.g., retrieval or k-NN agreement on unpaired batches) or substantially soften the claims about robustness.
minor comments (6)
- [Algorithm 1] Typo: 'LinearAlignement' should be 'LinearAlignment'.
- [Section 3.3] Typo: 'we fit a a simple linear alignment model' should read 'we fit a simple linear alignment model'.
- [Appendix C.2] Typo: 'permutatin' should be 'permutation' in the Monge formulation paragraph.
- [Table 4] The benchmark name 'Flickr30' should be 'Flickr30k' for consistency with the text and references.
- [Section 6 / Appendix A.1] The hyperparameters α, ε, ε*, and the CCA regularization λ are all prescribed as fixed values or selected on the CC3M validation set, but the selected α values for each experiment are not reported. The sensitivity statement 'comparable results can be obtained with alternative settings' would be more convincing with a small table or a figure showing performance versus α.
- [References] The reference list contains a formatting artifact: 'V ouitsis' should be 'Vouitsis'. Also, the arXiv identifier in the manuscript header 'arXiv:2602.23353v1' is not a standard format for the submission venue; this should be cleaned up.
Circularity Check
No significant circularity: SOTAlign's semi-supervised target is a pseudo-label from a linear teacher, but the central claims are evaluated on held-out benchmarks and the gradient result is a theorem.
full rationale
The derivation chain is not circular. The linear teacher (Eqs. 6-8) is fit on the small paired set and then used only to define the pseudo-label affinity K* on unpaired batches (Algorithm 1 line 6; Eq. 4). This is standard self-training/pseudo-labeling, not a prediction forced by definition: the reported gains are measured on held-out COCO/Flickr30k/ImageNet, which are not used to construct K*. If the linear-transfer premise fails, Table 7 shows performance degrades under distribution shift, which is exactly what an independent empirical assumption should do. The KLOT gradient (Theorem 5.1, Appendix C.2) is derived from entropic OT duality and the envelope theorem, not from fitted constants. The only self-citations are to the POT software library in implementation details and to an unrelated graph-matching paper in a list of Sinkhorn applications; neither is load-bearing for the central claim. The main limitations are empirical (small number of encoder pairs in the r=0.83 correlation; degradation under shift), not circular reasoning. The central results therefore have independent content and are not entailed by the paper's own inputs.
Assumptions & free parameters
free parameters (5)
- alpha (KLOT regularization weight) =
selected from {1e-3, 1e-4, 1e-5} based on CC3M validation retrieval
- epsilon (joint-space entropic regularization) =
0.05
- epsilon* (reference-space entropic regularization) =
0.01
- CCA regularization lambda =
0.1
- Sinkhorn iterations =
100
assumptions (5)
- domain assumption Platonic Representation Hypothesis / approximate linear compatibility of frozen unimodal encoders
- standard math Entropic OT dual representation: log OT_epsilon(K) = u 1^T + K/epsilon + 1 v^T
- standard math Envelope theorem applies to the strongly convex entropic OT problem
- domain assumption Cosine affinity between batch embeddings captures semantic geometry
- domain assumption Sinkhorn algorithm converges to OT_epsilon(K) within 100 iterations
Cite this review
Pith. "Pith review of SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport." pith.science (2026). https://pith.science/paper/YS4AO72Q
@misc{pith2026260223353,
author = {Pith},
title = {Pith review of: SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport},
year = {2026},
howpublished = {\url{https://pith.science/paper/YS4AO72Q}},
note = {Machine review of arXiv:2602.23353}
}
read the original abstract
The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at https://github.com/ExplainableML/SOTAlign.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 2 Pith papers
-
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
ToMA uses persistent homology on H0-death and lightweight H1-birth edges to align multimodal manifolds, delivering stable gains on remote sensing and consistent benefits on fashion retrieval.
-
MSAlign: Aligning Molecule and Mass Spectra Foundation Models for Metabolite Identification
MSAlign aligns frozen DreaMS and ChemBERTa models with MLPs and candidate-based contrastive learning to outperform prior methods on molecule retrieval from MS/MS spectra while quantifying distribution shift in data splits.
Reference graph
Works this paper leans on
-
[1]
Adams, R. P. and Zemel, R. S. Ranking via sinkhorn propa- gation.arXiv preprint arXiv:1106.1925,
arXiv 1925
-
[6]
Our approach significantly outperforms both alternatives in terms of memory usage and runtime. In particular, depending on the value of ϵ, which controls the number of Sinkhorn iterations (with convergence scaling as O(1/ϵ2)), the proposed method can be up to 100× more memory efficient than unrolling and up to50×faster than implicit differentiation. 14 Se...
2025
-
[8]
Jha, R., Zhang, C., Shmatikov, V ., and Morris, J. X. Har- nessing the universal geometry of embeddings.arXiv preprint arXiv:2505.12540,
-
[11]
Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,
-
[13]
Sim´eoni, O., V o, H. V ., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V ., Szafraniec, M., Yi, S., Rama- monjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., J´egou, H., Labatut, P., and Bojanowski, P. Dinov3.arXiv ...
-
[14]
Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y ., Mustafa, B., H ´enaff, O., Harmsen, J., Steiner, A., and Zhai, X. SigLIP 2: Multilingual vision- language encoders with improved semantic understand- ing, localization, and dense features.arXiv preprint arXiv:2502.14786,
-
[16]
Experimental Setting We outline the experimental setup in Section 6.3
11 Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport A. Experimental Setting We outline the experimental setup in Section 6.3. Here, we provide further details on our implementation and baselines. A.1. Implementation Details Following Zhang et al. (2025a), we create global image representations by concatenating the [CL...
2018
-
[17]
We mainly employ DINOv3 ViT-L (Sim´eoni et al.,
with a cosine annealing learning-rate schedule, a maximum learning rate of10 −4, and a weight decay of10 −5, and train for 2000 iterations. We mainly employ DINOv3 ViT-L (Sim´eoni et al.,
2000
Show all 27 references
-
[18]
as the pretrained vision and language encoders, respectively. In Section 6.2, we additionally evaluate SOTAlign with DINOv2 ViT-L (Oquab et al., 2024), Qwen3-Embedding-8B (Zhang et al., 2025b), and Llama-Embed-Nemotron-8B (Babakhin et al., 2025). All of these language models a...
2024
-
[19]
loss exclusively on paired data. This method represents a series of recent supervised contrastive methods for the alignment of pretrained unimodal vision and language models (V ouitsis et al., 2024; Maniparambil et al., 2025; Huang et al., 2025). Following Zhang et al. (2025a)...
2024
-
[20]
We further experiment with disjoint images and texts from CC12M (Changpinyo et al., 2021), COCO (Lin et al., 2014), ImageNet (Deng et al., 2009), and WikiText (Merity et al., 2016)
with both raw web captions and synthetic captions generated by DreamLIP (Zheng et al., 2024). We further experiment with disjoint images and texts from CC12M (Changpinyo et al., 2021), COCO (Lin et al., 2014), ImageNet (Deng et al., 2009), and WikiText (Merity et al., 2016). W...
2024
-
[21]
• Retrieval (T2I, I2T): COCO (Lin et al., 2014), Flickr30 (Plummer et al.,
2014
-
[24]
Our method achieves gains of +5.5 on COCO, +28.7 on Flickr30k, and +18.2 on Polyvore I2T R@5
In Table 11, we report the performance of SOTAlign adhering to their architectural choices. Our method achieves gains of +5.5 on COCO, +28.7 on Flickr30k, and +18.2 on Polyvore I2T R@5. However, if we lift these constraints and instead use linear alignment layers with a target...
2021
-
[25]
and without these constraints. COCO Flickr30k Polyvore 100 Pairs 500 Pairs 500 Pairs MethodI2T T2I I2T T2I I2T T2I CSA 1.3 1.0 1.3 0.8 1.3 1.0 Contrastive 8.5 5.8 9.5 9.8 13.8 11.5 SUE 21.5 18.3 19.8 22.0 22.8 20.8 SOTAlign(with SUE constraints) 27.0 28.8 48.5 48.8 41.0 39.8 S...
2004
-
[27]
replaces linear projections with angular projections and computes optimal transport on the circle, thereby respecting the intrinsic geometry of directional data. Total sliced Wasserstein distanceWe introduce the total spherical sliced Wasserstein distance d as a measure of the...
2019
-
[2008]
Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding.arXiv preprint arXiv:1807.03748,
-
[2009]
The quest for the graph level autoencoder (grale).arXiv preprint arXiv:2505.22109,
Krzakala, P., Melo, G., Laclau, C., d’Alch ´e Buc, F., and Flamary, R. The quest for the graph level autoencoder (grale).arXiv preprint arXiv:2505.22109,
-
[2014]
D., Shi, K., Shahbazi, A., Land- man, B
Liu, X., Bai, Y ., Mart´ın, R. D., Shi, K., Shahbazi, A., Land- man, B. A., Chang, C., and Kolouri, S. Linear spherical sliced optimal transport: A fast metric for comparing spherical data.arXiv preprint arXiv:2411.06055,
-
[2015]
We run as many Sinkhorn iterations as required to achieve marginal convergence within a tolerance of10 −6
13 Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport Figure 6.Comparison of memory usage, runtime, and number of Sinkhorn iterations for different gradient computation strategies. We run as many Sinkhorn iterations as required to achieve ma...
2018
-
[2017]
Huang, W., Wu, A., Yang, Y ., Luo, X., Yang, Y ., Hu, L., Dai, Q., Wang, C., Dai, X., Chen, D., Luo, C., and Qiu, L
ISBN 9781450349062. Huang, W., Wu, A., Yang, Y ., Luo, X., Yang, Y ., Hu, L., Dai, Q., Wang, C., Dai, X., Chen, D., Luo, C., and Qiu, L. Llm2clip: Powerful language model unlocks richer visual representation.arXiv preprint arXiv:2411.04997,
-
[2018]
Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., Gala, J., Siblini, W., Krzemi´nski, D., Winata, G. I., Sturua, S., Utpala, S., Ciancone, M., Schaeffer, M., Sequeira, G., Misra, D., Dhakal, S., Rystrøm, J., Solomatin, R., ¨Omer C ¸a˘gatan, Kundu, A., B...
-
[2019]
for more details. Letn∈Nand denote byP n the set of permutation matrices, Pn ={P∈ {0,1} n×n |P1=1, P ⊤1=1},(Permutations Matrices) and byΠ n the set of bistochastic matrices, Πn ={T∈R n×n + |T1=1, T ⊤1=1}.(Transport Plans) We further define the (negative) entropy of a transpor...
2019
-
[2020]
Reliability of cka as a similarity measure in deep learning.arXiv preprint arXiv:2210.16156,
Davari, M., Horoi, S., Natik, A., Lajoie, G., Wolf, G., and Belilovsky, E. Reliability of cka as a similarity measure in deep learning.arXiv preprint arXiv:2210.16156,
-
[2022]
and Ranka, S
Emami, P. and Ranka, S. Learning permutations with sinkhorn policy gradient.arXiv preprint arXiv:1805.07010,
-
[2023]
Assessing and learn- ing alignment of unimodal vision and language models
Zhang, L., Yang, Q., and Agrawal, A. Assessing and learn- ing alignment of unimodal vision and language models. InCVPR, pp. 14604–14614, 2025a. Zhang, Y ., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al. Qwen3 embed- ding: Advancing ...
-
[2024]
Babakhin, Y ., Osmulski, R., Ak, R., Moreira, G., Xu, M., Schifferer, B., Liu, B., and Oldridge, E
URL https://huguesva.github.io/blog/2024/ inverseOT_mongegap/. Babakhin, Y ., Osmulski, R., Ak, R., Moreira, G., Xu, M., Schifferer, B., Liu, B., and Oldridge, E. Llama- embed-nemotron-8b: A universal text embedding model for multilingual and cross-lingual tasks.arXiv preprint...
2024
-
[2025]
Wasserstein discriminant analysis.Machine Learning, 107(12):1923–1945,
Flamary, R., Cuturi, M., Courty, N., and Rakotomamonjy, A. Wasserstein discriminant analysis.Machine Learning, 107(12):1923–1945,
1923
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.