REVIEW 3 major objections 4 minor 1 cited by
LSMI-Sinkhorn: Semi-supervised Mutual Information Estimation with Optimal Transport
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Squared-loss mutual information can be estimated from a few data pairs plus many unpaired samples, by learning an optimal-transport alignment and fitting a density-ratio model.
desk verdict A clean semi-supervised extension of LSMI via Sinkhorn coupling, with a real self-training bias concern that needs analysis before the estimates are trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the coupling matrix $\Pi$ with $\Pi\mathbf{1}_{n_y}=n_x^{-1}\mathbf{1}_{n_x}$ and $\Pi^\top\mathbf{1}_{n_x}=n_y^{-1}\mathbf{1}_{n_y}$, learned by Sinkhorn iterations; it converts unpaired marginal samples into a soft surrogate for the joint density. The objective couples $\Pi$ with the density-ratio parameter $\alpha$ through the vector $h_{\Pi,\beta}=\frac{\beta}{n}\sum_i \phi(x_i,y_i)+(1-\beta)\sum_{i,j}\pi_{ij}\phi(x'_i,y'_j)$, so that fixing either variable makes the other subproblem convex. This alternating structure, together with the rank-$b$ cost matrix $C_\alpha=K^\top\mathrm{diag}(\alpha)L$, gives the monotone-decrease guarantee and the $O(n_xn_y)$ iteration cost.
What would settle it
On synthetic data with a known SMI value (for example $Y=0.5X+\mathcal{N}(0,0.01)$ with $X$ standard normal), compute the ground-truth SMI from a very large paired sample, then run LSMI-Sinkhorn with $n=10$ true pairs and $n_x=n_y=1000$ unpaired marginal samples; if the estimate remains far from ground truth while the learned $\Pi$ is sharply concentrated on a visibly wrong matching, the surrogate-coupling assumption is refuted.
Extended reading notes
Core claim
The central claim is that estimating SMI from limited paired data can be cast as minimizing the squared error between the true density ratio $r(x,y)=p(x,y)/(p(x)p(y))$ and a kernel model $r_\alpha(x,y)=\alpha^\top \phi(x,y)$, with the joint term in the loss approximated by a weighted mix of the few true pairs and a coupling $\Pi$ over unpaired samples. The authors solve $\min_{\Pi,\alpha} J(\Pi,\alpha)$ under marginal constraints on $\Pi$, alternating between a closed-form least-squares update $\hat\alpha=(H+\lambda I)^{-1}h_{\Pi,\beta}$ and a Sinkhorn update for $\Pi$. This yields a monotonically decreasing objective, $O(n_xn_y)$ per-iteration cost, and, in the experiments, SMI values close to those of a full-sample estimator while pair-only and Gromov-Wasserstein-based baselines deviate substantially. The paper presents this as the first semi-supervised SMI estimator, and demonstrates downstream use in deep image matching and rectangular photo album summarization.
Load-bearing premise
The load-bearing premise is that the learned coupling matrix $\Pi$ is a valid stand-in for the true joint density $p(x,y)$ over the unpaired samples; if Sinkhorn converges to a coupling that is well matched to the density-ratio model but not to the real correspondence, the estimated SMI can be systematically wrong, and the paper offers no consistency or bias analysis to rule this out.
Editorial extensions
If this is right
- With only tens of paired samples and abundant unpaired marginals, SMI values close to a full-sample estimate can be obtained on synthetic and UCI benchmarks.
- The alternating scheme decreases the objective monotonically and converges in a few iterations, making the estimator stable enough to be used as a training objective.
- Per-iteration cost $O(n_xn_y)$ makes the method feasible for large unpaired sets, with GPU-accelerated runs on 10,000 unpaired samples taking seconds.
- Because it allows $n_x\neq n_y$, the method extends matching and summarization to rectangular layouts such as 320 images into a $16\times 20$ grid, where square-only methods fail.
- The learned coupling is concentrated along the diagonal for dependent data and uniform for independent data, providing a direct visual diagnostic of dependence.
Reading between the lines
- Editorial extension: if the coupling surrogate is reliable, the same plug-in trick should extend to other $f$-divergences or Wasserstein dependency measures wherever unpaired marginals are abundant.
- Editorial extension: because the convergence proof only shows monotone decrease of $J$, not convergence of $\Pi$ to the true joint, a stress test with deliberately misspecified kernels or badly chosen basis vectors would reveal whether the estimator's accuracy comes from the coupling or mainly from the paired-sample term.
- Editorial extension: the rank constraint on $C_\alpha$ suggests that random-feature or Nystr\"om approximations of the kernel could scale the method well beyond the 10,000-sample demonstrations reported.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript addresses the problem of estimating squared-loss mutual information (SMI) when only a small number of paired samples and many unpaired samples are available. It introduces a density-ratio model r_α(x,y)=α^T φ(x,y), approximates the cross term ∫ r_α p(x,y) by a weighted combination of paired samples and unpaired samples aligned through a coupling matrix Π, and formulates a joint optimization problem (Eq. 5) regularized by entropy on Π and L2 on α. The proposed LSMI-Sinkhorn algorithm alternates between a closed-form α update (Eq. 8) and a Sinkhorn update for Π (Eq. 6); Proposition 1 proves monotone decrease of the surrogate objective. Experiments on synthetic data, UCI datasets, deep image matching, and photo album summarization are reported to support the claim that SMI can be estimated from tens of pairs plus unpaired marginals.
Significance. If the estimation claim holds, the paper makes a useful contribution: it is, to my knowledge, the first semi-supervised SMI estimator, it avoids the O(n^3) Hungarian matching by using Sinkhorn (O(nx ny)), it comes with code, and its alternating scheme is simple and monotonically decreasing. The applications to image matching and photo album summarization broaden the potential impact beyond MI estimation. However, the statistical validity of the estimator is the load-bearing issue: because Π is learned from the same density-ratio cross-term it is meant to replace, the manuscript needs a bias/consistency argument or a convincing repeated-run bias experiment before the 'can estimate SMI from few pairs' claim can be accepted. The current single-run plots do not provide such evidence.
major comments (3)
- [§3.1, Eqs. (4)–(5); §3.2, Proposition 1] The central statistical claim is not supported because the coupling matrix Π is learned by maximizing the same density-ratio cross-term it is introduced to estimate. For fixed α, the Sinkhorn subproblem in Eq. (6) is equivalent to maximizing Σ_ij π_ij α^T φ(x'_i,y'_j), so mass is concentrated on pairs with large current density-ratio values; the subsequent α update in Eq. (7) then fits r_α to this self-selected mixture. Proposition 1 proves only monotone decrease of the surrogate J, not convergence of Π to the true joint density, nor unbiasedness or consistency of \hat{SMI}. The 'ideal' discussion in Eq. (4) also presumes a latent true pairing of the unpaired samples. The manuscript should add either a consistency/bias analysis or, at minimum, a bias experiment on synthetic data with known SMI (including the independent case) with repeated trials before claiming that SMI can be estimated from tens of paired samples.
- [§5.3, Figures 3, 5, and 6] The empirical evidence does not rule out a shared upward bias. The SMI estimation plots report single runs without error bars or repeated trials, and the 'Random' panels show small but apparently positive SMI values. Moreover, 'LSMI (full)' is itself an estimate from 10,000 paired samples and is treated as ground truth in both synthetic and UCI experiments. With single runs, the closeness of LSMI-Sinkhorn to LSMI (full) could simply mean that both estimators are biased in the same direction. Please report mean and standard deviation over multiple random data splits, and where possible compare to an analytic or high-accuracy numerical SMI value.
- [§3.3; §5.1] The role of the entropic regularization ϵ in the statistical behavior of the estimator is not analyzed. The paper fixes ϵ=0.3 without a sensitivity study, and the ϵ→0 limit of Eq. (6) drives Π toward a hard assignment, reproducing LSOM, which the authors themselves describe as a matching method rather than a density estimator. Since no consistency result is provided for the regularized estimator, the influence of ϵ on the bias-variance trade-off of \hat{SMI} should be investigated analytically or empirically; otherwise the choice of ϵ appears arbitrary for the estimation claim.
minor comments (4)
- [§5.5] The photo album summarization description and Figures 8 and 9 are duplicated verbatim in the manuscript; please remove the duplicate passages.
- [§3.2, after Eq. (8)] The text states that H is a positive definite matrix, but as a sum of outer products H is positive semidefinite; only H + λI is guaranteed to be positive definite. Please correct this wording.
- [§5.4, Figure 7] The matching-accuracy curves are shown without error bars, and the text does not specify how many random shuffles were averaged or how exactly top-1 and top-2 accuracies are defined; please clarify.
- [Throughout] There are small typos such as 'Gromove-Wasserstein' in §3.3, and the related-work discussion of LSOM would benefit from consistently citing the same references ([22] vs [24]) for the original algorithm.
Circularity Check
No significant circularity: the reported SMI is a different functional of the fitted density ratio, not the optimized objective renamed, and the Pi-alpha alternation is a latent-variable estimation loop rather than a definitional reduction.
full rationale
The derivation chain is: SMI is defined through the density ratio r(x,y) in Eq. (1); r is modeled by r_alpha in Eq. (2); the squared-error loss in Eq. (3) is the estimation objective; because paired samples are scarce, the cross-expectation over p(x,y) is approximated in Eq. (4) by a mixture of the observed pairs and a Pi-weighted combination of unpaired pairs; then Pi and alpha are jointly optimized in Eq. (5) by alternating Sinkhorn updates (Eq. (6)) and ridge-regression updates (Eqs. (7)-(8)). The final reported SMI is computed as the sample average of (r_alpha - 1)^2 over all combinations, which is not equal to the optimized objective J. Concretely, J contains the Pi-weighted cross-term h_{Pi,beta} plus regularizers, while the plug-in SMI estimate uses the unweighted empirical average h_emp; these are different functionals. Thus no fitted parameter is renamed as a prediction. The mutual dependence of Pi and alpha is a self-consistent latent-alignment procedure, analogous to EM, and any resulting bias is a statistical consistency concern, not a definitional circularity. The paper cites prior LSOM work only as a special-case baseline, and it cites Sinkhorn, Cuturi, and Suzuki et al. for standard external tools; no load-bearing argument reduces to a self-citation or to an author-imported uniqueness theorem. The lack of bias or consistency analysis for Pi is a genuine weakness, and the empirical evaluation uses LSMI(full) as a surrogate ground truth, but these are correctness risks rather than circularity in the derivation chain.
Assumptions & free parameters
free parameters (4)
- lambda (regularization on alpha) =
selected by cross-validation from {0.1, 0.01, 0.001, 0.0001}
- beta (paired/unpaired balance) =
selected by cross-validation from {0.2, 0.4, 0.6, 0.8, 1.0}
- epsilon (entropic regularization) =
0.3 (fixed)
- b (number of basis vectors) =
200 (fixed)
assumptions (4)
- domain assumption The linear density-ratio model r_alpha(x,y)=alpha^T phi(x,y) is flexible enough to approximate the true ratio.
- ad hoc to paper The transport coupling Pi learned from the objective is a valid empirical surrogate for the joint density p(x,y) on unpaired samples.
- domain assumption Unpaired samples are i.i.d. draws from the marginals and are independent of the paired set.
- standard math The Sinkhorn algorithm converges to the entropy-regularized optimal transport solution.
Cite this review
Pith. "Pith review of LSMI-Sinkhorn: Semi-supervised Mutual Information Estimation with Optimal Transport." pith.science (2026). https://pith.science/paper/TJWMRSQT
@misc{pith2026190902373,
author = {Pith},
title = {Pith review of: LSMI-Sinkhorn: Semi-supervised Mutual Information Estimation with Optimal Transport},
year = {2026},
howpublished = {\url{https://pith.science/paper/TJWMRSQT}},
note = {Machine review of arXiv:1909.02373}
}
abstract
Estimating mutual information is an important statistics and machine learning problem. To estimate the mutual information from data, a common practice is preparing a set of paired samples $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^n \stackrel{\mathrm{i.i.d.}}{\sim} p(\mathbf{x},\mathbf{y})$. However, in many situations, it is difficult to obtain a large number of data pairs. To address this problem, we propose the semi-supervised Squared-loss Mutual Information (SMI) estimation method using a small number of paired samples and the available unpaired ones. We first represent SMI through the density ratio function, where the expectation is approximated by the samples from marginals and its assignment parameters. The objective is formulated using the optimal transport problem and quadratic programming. Then, we introduce the Least-Squares Mutual Information with Sinkhorn (LSMI-Sinkhorn) algorithm for efficient optimization. Through experiments, we first demonstrate that the proposed method can estimate the SMI without a large number of paired samples. Then, we show the effectiveness of the proposed LSMI-Sinkhorn algorithm on various types of machine learning problems such as image matching and photo album summarization. Code can be found at https://github.com/csyanbin/LSMI-Sinkhorn.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
SiamJEPA: On the Role of Siamese Student Encoders in JEPA
SiamJEPA, a masked-image JEPA variant with Siamese student encoders and an EMA teacher, improves ImageNet linear probing accuracy over a JEPA-like baseline and beats MAE at 400 epochs.
Reference graph
Works this paper leans on
-
[1]
Belghazi, M.I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Hjelm, D., Courville, A.: Mutual information neural estimation. In: ICML (2018)
work page 2018
-
[2]
In: ICML (2019) 16 Liu Y., Yamada M., Tsai YH., Le T., Salakhutdinov R., Yang Y
Bunne, C., Alvarez-Melis, D., Krause, A., Jegelka, S.: Learning generative models across incomparable spaces. In: ICML (2019) 16 Liu Y., Yamada M., Tsai YH., Le T., Salakhutdinov R., Yang Y
work page 2019
-
[3]
Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: AISTATS (2011)
work page 2011
-
[4]
John Wiley & Sons, Inc., Hoboken, NJ, USA, 2nd edn
Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, Inc., Hoboken, NJ, USA, 2nd edn. (2006)
work page 2006
-
[5]
Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: NIPS (2013)
work page 2013
-
[6]
Djuric, N., Grbovic, M., Vucetic, S.: Convex kernelized sorting. In: AAAI (2012)
work page 2012
-
[7]
Flamary, R., Courty, N.: Pot python optimal transport library (2017), https: //github.com/rflamary/POT
work page 2017
-
[8]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014)
work page 2014
Show all 26 references
-
[9]
In: ICLR (2019)
Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., Bengio, Y.: Learning deep representations by mutual information estimation and maximization. In: ICLR (2019)
2019
-
[10]
Naval Research Logistics Quarterly 2(1-2), 83–97 (1955)
Kuhn, H.: The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2(1-2), 83–97 (1955)
1955
-
[11]
Foundations of Computational Mathematics 11(4), 417–487 (2011)
M´ emoli, F.: Gromov–wasserstein distances and the metric approach to object matching. Foundations of Computational Mathematics 11(4), 417–487 (2011)
2011
-
[12]
NeurIPS (2019)
Ozair, S., Lynch, C., Bengio, Y., Oord, A.v.d., Levine, S., Sermanet, P.: Wasserstein dependency measure for representation learning. NeurIPS (2019)
2019
-
[13]
Foundations and Trends® in Machine Learning 11(5-6), 355–607 (2019)
Peyr´ e, G., Cuturi, M.: Computational optimal transport. Foundations and Trends® in Machine Learning 11(5-6), 355–607 (2019)
2019
-
[14]
In: ICML (2016)
Peyr´ e, G., Cuturi, M., Solomon, J.: Gromov-wasserstein averaging of kernel and distance matrices. In: ICML (2016)
2016
-
[15]
IEEE transactions on pattern analysis and machine intelligence 32, 1809–1821 (2010)
Quadrianto, N., Smola, A., Song, L., Tuytelaars, T.: Kernelized sorting. IEEE transactions on pattern analysis and machine intelligence 32, 1809–1821 (2010)
2010
-
[16]
SIAM Journal on Scientific Computing 41(3), A1443–A1481 (2019)
Schmitzer, B.: Stabilized sparse scaling algorithms for entropy regularized trans- port problems. SIAM Journal on Scientific Computing 41(3), A1443–A1481 (2019)
2019
-
[17]
Proceedings of the American Mathematical Society 45(2), 195–198 (1974)
Sinkhorn, R.: Diagonal equivalence to matrices with prescribed row and column sums. Proceedings of the American Mathematical Society 45(2), 195–198 (1974)
1974
-
[18]
In: NIPS (2009)
Sriperumbudur, B.K., Fukumizu, K., Gretton, A., Lanckriet, G.R., Sch¨ olkopf, B.: Kernel choice and classifiability for rkhs embeddings of probability distributions. In: NIPS (2009)
2009
-
[19]
In: AISTATS (2010)
Suzuki, T., Sugiyama, M.: Sufficient dimension reduction via squared-loss mutual information estimation. In: AISTATS (2010)
2010
-
[20]
BMC Bioin- formatics 10(S52) (2009)
Suzuki, T., Sugiyama, M., Kanamori, T., Sese, J.: Mutual information estimation reveals global associations between stimuli and biological processes. BMC Bioin- formatics 10(S52) (2009)
2009
-
[21]
In: ISIT (2009)
Suzuki, T., Sugiyama, M., Tanaka, T.: Mutual information approximation via max- imum likelihood estimation of density ratio. In: ISIT (2009)
2009
-
[22]
IEEE transactions on Pat- tern Analysis and Machine Intelligence 37(9), 1764–1776 (2015)
Yamada, M., Sigal, L., Raptis, M., Toyoda, M., Chang, Y., Sugiyama, M.: Cross- domain matching with squared-loss mutual information. IEEE transactions on Pat- tern Analysis and Machine Intelligence 37(9), 1764–1776 (2015)
2015
-
[23]
In: AAAI (2010)
Yamada, M., Sugiyama, M.: Dependence minimizing regression with model selec- tion for non-linear causal inference under non-gaussian noise. In: AAAI (2010)
2010
-
[24]
In: AISTATS (2011)
Yamada, M., Sugiyama, M.: Cross-domain object matching with model selection. In: AISTATS (2011)
2011
-
[25]
In: IJCAI (2018)
Yan, Y., Li, W., Wu, H., Min, H., Tan, M., Wu, Q.: Semi-supervised optimal transport for heterogeneous domain adaptation. In: IJCAI (2018)
2018
-
[26]
In: AAAI (2019)
Zhao, S., Song, J., Ermon, S.: Infovae: Balancing learning and inference in varia- tional autoencoders. In: AAAI (2019)
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.