A note on the relations between mixture models, maximum-likelihood and entropic optimal transport

Etienne Lasalle (OCKHAM); Titouan Vayer (OCKHAM)

REVIEW 2 major objections 3 minor 7 references

A note on the relations between mixture models, maximum-likelihood and entropic optimal transport

T0 review · 2 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Maximum-likelihood estimation for mixture models is equivalent to minimizing an entropic optimal transport problem at regularization strength $\varepsilon = 1$.

desk verdict A clean, self-contained proof of a known equivalence, undermined by a concrete factor-n error in the claimed EM-as-block-coordinate-descent illustration. read the letter →

arxiv 2501.12005 v2 pith:LWSGSHUB submitted 2025-01-21 stat.ML cs.LG

Titouan Vayer (OCKHAM) , Etienne Lasalle (OCKHAM) This is my paper · ORCID

classification stat.MLcs.LG MSC 62F1062H3049Q2262B10

keywords mixturemodelsmaximumlikelihoodestimationentropicoptimaltransportEMalgorithmGaussianKullback-Leiblerdivergencelog-sum-expidentityblock-coordinatedescent

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that maximum-likelihood estimation in a discrete mixture model is the same optimization as minimizing an entropic optimal transport problem, with the transport cost between sample $i$ and component $j$ equal to the negative log of the conditional density. Concretely, the normalized negative log-likelihood equals a semi-relaxed entropic OT objective, and after optimizing the mixing weights it equals the full entropic OT value with regularization strength $\varepsilon = 1$. The note then shows that the standard EM algorithm for Gaussian mixtures is exactly a block-coordinate descent on this transport loss. The presentation is deliberately pedagogical and relies on the variational log-sum-exp identity plus an information decomposition of the KL divergence.

What carries the argument

The engine is the Gibbs variational principle, or log-sum-exp identity: $\log\left(\sum_j \pi_j e^{h_j}\right) = \max_{p \in \Delta_K} \sum_j h_j p_j - \mathrm{KL}(p \| \pi)$, with optimal $p_j \propto \pi_j e^{h_j}$. Applied to each sample, this rewrites the log-likelihood as a minimization over the rows of a transport plan. The second ingredient is the KL chain rule $\mathrm{KL}(P \mid a b^\top) = \mathrm{KL}(P \mid (P\mathbf{1})(P^\top\mathbf{1})^\top) + \mathrm{KL}(P\mathbf{1} \| a) + \mathrm{KL}(P^\top\mathbf{1} \| b)$, which lets the author replace the semi-relaxed constraint by a full coupling constraint once $\pi$ is optimized, producing the entropic OT objective at $\varepsilon = 1$.

What would settle it

Take a Gaussian mixture with two components and set one component's covariance to be degenerate at a sample location not equal to its mean; then that component's conditional density is zero at the sample, so $C_{ij}(\theta)$ is infinite. Substituting into equation (2.6), the right-hand side is undefined under the paper's stated arithmetic while the left-hand side remains finite because the other component covers the point. Checking whether Proposition 2.3 survives this case with an extended-value convention would settle the exact scope of the identity.

Watch

Extended reading notes

Core claim

The central claim is the three-part identity in Proposition 2.3. For any discrete mixture model and any $n$ i.i.d. samples, writing $C_{ij}(\theta) = -\log P_{X|Y}(x_i|j,\theta)$, the normalized negative log-likelihood satisfies $(1/n)L(\pi,\theta) = \min_{P \in \mathcal{U}_K(\mathbf{1}_n/n)} \langle C(\theta), P \rangle + \mathrm{KL}(P \mid (\mathbf{1}_n/n)\pi^\top)$; this quantity is bounded above by $\mathrm{OT}_{\varepsilon=1}(\mathbf{1}_n/n, \pi, C(\theta))$; and after minimizing over $\pi$ the upper bound becomes an equality, so $\min_{\pi,\theta}(1/n)L(\pi,\theta) = \min_{\pi,\theta} \mathrm{OT}_{\varepsilon=1}(\mathbf{1}_n/n, \pi, C(\theta))$. Since the EM updates for a Gaussian mixture—posterior weights, class proportions, means, and covariance—are obtained by minimizing the same loss over each block of variables in turn, the note identifies EM as block-coordinate descent on an entropic OT objective.

Load-bearing premise

All component densities must be strictly positive at every observed sample, because the proof defines the transport cost as the negative logarithm of those densities and explicitly leaves the $0 \log 0$ edge case unspecified.

Editorial extensions

If this is right

For any fixed $\theta$, minimizing the negative log-likelihood over the mixing weights $\pi$ is the same as minimizing the entropic OT value $\mathrm{OT}_{\varepsilon=1}(\mathbf{1}_n/n, \pi, C(\theta))$ over $\pi$.
The standard EM algorithm for Gaussian mixtures—E-step posterior weights, M-step proportions, means, and covariance—is a block-coordinate descent on the entropic OT loss in equation (3.1).
The normalized negative log-likelihood is always bounded above by the entropic OT objective, and optimizing $\pi$ closes the gap, so the upper bound is tight at the optimum.
The equivalence extends to infinite mixtures, where the entropic OT problem is posed between the latent measure and the empirical data measure with cost $c(x,y;\theta) = -\log P_{X|Y}(x|y,\theta)$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the equivalence holds only at $\varepsilon = 1$, varying the regularization strength in the entropic OT problem defines a family of objectives that interpolate between maximum likelihood and pure optimal transport; the note does not explore this direction, but it suggests a natural tempering or robustness knob for mixture estimation.
The row-update formula for $P$ in equation (3.2) is exactly the E-step, so any convergent approximation to that semi-relaxed entropic OT problem—for instance a truncated Sinkhorn iteration—could be substituted into the block-coordinate loop and would still describe a valid EM-like algorithm with the same fixed points.
The identity reinterprets likelihood-based comparison of mixture models as comparison of entropic OT costs, which could allow model-selection criteria to be expressed directly through transport-plan costs; the note does not develop this consequence.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, what you should know: the main equivalence (Prop 2.3) is correct, self-contained, and clearly presented, but the Section 3 EM illustration contains a concrete factor-n error in the covariance update. That error is repairable, but as written the claim that EM is a block-coordinate descent on the EOT loss is not demonstrated.

The note does what it says: it proves, in a discrete setting, that minimizing the negative log-likelihood of a mixture model is equivalent to minimizing an entropic OT problem, and that minimizing over π turns the semi-relaxed formulation into the standard coupling formulation. Lemmas 2.2 and 5.2 are proved cleanly, and the logic of the argument is easy to follow. The authors are upfront that the result is already known and cite Rigollet-Weed, Mena et al., and Diebold et al. So no mathematical novelty is claimed, and that's fine for a pedagogical note.

The main soft spot is the GMM illustration. In Section 3, the loss being minimized is (3.1), with cost C_{ij}(θ) = -log P_{X|Y}(x_i|j,θ). For the Gaussian, that includes (1/2) log det Σ per term. Since sum_{ij} P_{ij}=1 for P in U_K(1/n 1_n), the total coefficient on log det Σ is 1/2, not n/2 as written in the manuscript. The correct stationary condition is Σ = sum_{ij} P_{ij}(x_i-μ_j)(x_i-μ_j)^T, which, with P_{ij}=r_{ij}/n, is exactly the standard EM tied-covariance update. The paper instead writes (3.4) with a 1/n factor in front of the sum, which gives (1/n^2) sum r_{ij}(...) and is wrong. The π and μ updates are correct. This error doesn't affect the main proposition, but it does undermine the advertised illustration. The fix is straightforward: replace n/2 by 1/2 and (3.4) by the unnormalized sum.

Also, the 0 log 0 edge case is waved away with a promise of future rigor. That's a minor issue given the stated positivity assumption; I wouldn't hang the paper on it.

Who gets value: someone who wants a compact derivation of the MLE-EOT equivalence, or an example of using the Gibbs variational principle in this setting. It's a decent teaching note. But it needs the covariance fix before it can be recommended as-is. A serious referee should be able to catch this and ask for revision, so yes, it deserves peer review. I would not cite it in my own work in its current form.

Reading group: maybe, if the coordinator wants to discuss the equivalence and the subtlety of getting the scaling right in BCD updates.

Referee Report

2 major / 3 minor

Summary. This note establishes a bridge between maximum-likelihood estimation for discrete mixture models and entropic optimal transport. After recalling the Gibbs variational principle, the authors rewrite the negative log-likelihood as a semi-relaxed entropic OT objective with cost C_ij(θ) = -log P_{X|Y}(x_i|j,θ), prove a decomposition of the KL divergence that yields a coupling constraint when the mixing weights are optimized, and conclude that the minimum of the normalized NLL over (π,θ) equals the minimum of the entropic OT problem over the same parameters. Section 3 illustrates the result on Gaussian mixtures and claims that the EM updates are exactly a block-coordinate descent on the EOT loss.

Significance. The main equivalence is not new and is properly attributed to Rigollet and Weed (2018) and related work, but the note's contribution is pedagogical: the proof is self-contained and elementary, relying only on Lemma 2.2 and the KL decomposition in Lemma 5.2, and it cleanly explains the role of the second marginal. The derivation is parameter-free in the sense that no quantities are fitted and no external numerical results are needed. The advertised GMM illustration is a useful bridge between EM and computational OT, provided the covariance update is corrected. With that fix, the note would be a concise and accessible reference for researchers working at the interface of mixture modeling and entropic OT.

major comments (2)

[Section 3, Eq. (3.4)] The covariance update is incorrect by a factor of n. After fixing P and π, the objective to minimize over θ is (1/2)∑_{ij}(x_i-μ_j)^TΣ^{-1}(x_i-μ_j)P_ij + (1/2)log det Σ (up to constants), because ∑_{ij}P_ij=1 for P∈U_K((1/n)1_n). The paper writes (n/2)log det Σ. Setting the derivative to zero then gives Σ = ∑_{ij} P_ij (x_i-μ_j)(x_i-μ_j)^T, not Σ = (1/n)∑_{ij}P_ij(...). In terms of the responsibilities r_ij=nP_ij, the correct update is the standard EM formula Σ=(1/n)∑_{ij}r_ij(...), while the printed (3.4) would give (1/n^2)∑_{ij}r_ij(...). For K=1 this is (1/n^2)∑(x_i-μ)^2 instead of (1/n)∑(x_i-μ)^2. This invalidates the claim that the updates 'exactly correspond' to EM; the error is localized and repairable by changing the coefficient to 1/2 and (3.4) to the unnormalized sum.
[Notations and Proposition 2.3] The main theorem is stated without any support condition on the component densities, but the proof uses log(P_{X|Y}(x_i|j,θ)) and KL divergences without defining 0 log 0 or ∞ costs. As the Notations paragraph admits, edge cases are deliberately left vague. To make Proposition 2.3 rigorous as stated, the authors should either assume P_{X|Y}(x_i|j,θ)>0 for all i,j,θ in the admissible set, or provide the extended-real argument that allows zero densities by restricting P_ij to the support of the density. This is a genuine gap in the central derivation, though it does not appear to affect the substance of the result under standard conventions.

minor comments (3)

[Section 3, Eq. (3.2)] The density in the displayed posterior is written P_{X|Y}(x|j,θ); it should be P_{X|Y}(x_i|j,θ) to match the i-th sample.
[Section 2, discussion after Eq. (2.1)] The remark 'in ⋆⋆ we used the Bayes formula' is slightly imprecise: the equality P_{X,Y}(x_i,j|β)=π_jP_{X|Y}(x_i|j,θ) is the product rule/definition of conditional probability rather than Bayes' theorem. Consider rewording.
[Abstract and Proposition 2.3] The abstract's phrase 'equivalent to minimizing over the parameters an optimal transport problem' is made precise only in Proposition 2.3 as an equality of minima; stating this explicitly in the abstract or introduction would avoid possible over-reading about equivalence of argmin sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core equivalence is derived from self-contained lemmas, and no fitted input or self-citation carries the argument.

full rationale

The paper's central claim—that MLE for a mixture model is equivalent to minimizing an entropic optimal transport problem—is established by an internal derivation chain. Lemma 2.2 (Gibbs variational principle) and Lemma 5.2 (KL decomposition) are both proved in Section 5 of the paper itself, and they are the only substantive tools used to transform the negative log-likelihood into a semi-relaxed EOT objective and then, after minimizing over π, into a full EOT objective (Proposition 2.3). The derivation is algebraic and does not rely on any external fitted parameters or on prior work by the same authors as a load-bearing assumption. Citations to Rigollet and Weed, Mena et al., and Cuturi are contextual and point to independent prior results (e.g., entropic OT regularization, existing relations) rather than being used to justify the present equivalence without proof. The GMM illustration in Section 3 contains a concrete mathematical error: the coefficient of log det Σ in the θ-update is written as n/2 instead of 1/2, leading to an erroneous covariance update (3.4). This is a correctness flaw that would break the claimed EM-as-BCD illustration, but it is not circularity: the flawed update is derived from the paper's own EOT loss, not from the target EM update, and the error is repairable without changing the main equivalence. Because there is no step where the conclusion is assumed in the premises, no fitted value is renamed as a prediction, and no self-citation chain is invoked, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters or invented entities; the note relies only on standard variational identities and the stated mixture model assumptions.

assumptions (5)

standard math Gibbs variational principle / dual representation of log-sum-exp (Lemma 2.2)
Used to rewrite each term of the negative log-likelihood as a maximization over the simplex; proved in Section 5.
standard math KL divergence decomposition for matrices (Lemma 5.2)
Used to relate the semi-relaxed objective to the full entropic OT problem with target marginal pi; proved in Section 5.
domain assumption Mixture model generative process with discrete latent variable and parametrized conditional densities P_{X|Y}(x|j,theta)
The equivalence is stated for this model class (Definition 2.1); the cost matrix is defined via these densities.
domain assumption Sample points x_i are i.i.d. and represented by the uniform empirical measure 1/n sum delta_{x_i}
The row marginal of P is fixed to 1/n 1_n; this is what makes the semi-relaxed plan formulation exact.
domain assumption Strictly positive densities and probabilities so that logs and KL divergences are finite
Needed to avoid 0 log 0; the note flags this vagueness in the Notations section.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A note on the relations between mixture models, maximum-likelihood and entropic optimal transport." pith.science (2026). https://pith.science/paper/LWSGSHUB

@misc{pith2026250112005,
  author       = {Pith},
  title        = {Pith review of: A note on the relations between mixture models, maximum-likelihood and entropic optimal transport},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LWSGSHUB}},
  note         = {Machine review of arXiv:2501.12005}
}

read the original abstract

This note aims to demonstrate that performing maximum-likelihood estimation for a mixture model is equivalent to minimizing over the parameters an optimal transport problem with entropic regularization. The objective is pedagogical: we seek to present this already known result in a concise and hopefully simple manner. We give an illustration with Gaussian mixture models by showing that the standard EM algorithm is a specific block-coordinate descent on an optimal transport loss.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages

[1]

Entropic optimal transport is maximum-likelihood deconvolution

Philippe Rigollet and Jonathan Weed. Entropic optimal transport is maximum-likelihood deconvolution. Comptes Rendus. Math \'e matique , 356 0 (11-12): 0 1228--1235, 2018

work page 2018
[2]

Sinkhorn em: an expectation-maximization algorithm based on entropic optimal transport

Gonzalo Mena, Amin Nejatbakhsh, Erdem Varol, and Jonathan Niles-Weed. Sinkhorn em: an expectation-maximization algorithm based on entropic optimal transport. arXiv preprint arXiv:2006.16548, 2020

arXiv 2006
[3]

A unified framework for hard and soft clustering with regularized optimal transport, 2024

Jean-Frédéric Diebold, Nicolas Papadakis, Arnaud Dessein, and Charles-Alban Deledalle. A unified framework for hard and soft clustering with regularized optimal transport, 2024. URL https://arxiv.org/abs/1711.04366

arXiv 2024
[4]

Computational optimal transport: With applications to data science

Gabriel Peyr \'e , Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11 0 (5-6): 0 355--607, 2019

2019
[5]

Sinkhorn distances: Lightspeed computation of optimal transport

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Neural Information Processing Systems (NeurIPS), 26, 2013

work page 2013
[6]

Machine learning: a probabilistic perspective

Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012

2012
[7]

Courty, and Valentin Emyia

Remi Flamary, Cédric Févotte, N. Courty, and Valentin Emyia. Optimal spectral transportation with application to music transcription. In Neural Information Processing Systems (NeurIPS), 2016

work page 2016

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Entropic optimal transport is maximum-likelihood deconvolution

Philippe Rigollet and Jonathan Weed. Entropic optimal transport is maximum-likelihood deconvolution. Comptes Rendus. Math \'e matique , 356 0 (11-12): 0 1228--1235, 2018

work page 2018

[2] [2]

Sinkhorn em: an expectation-maximization algorithm based on entropic optimal transport

Gonzalo Mena, Amin Nejatbakhsh, Erdem Varol, and Jonathan Niles-Weed. Sinkhorn em: an expectation-maximization algorithm based on entropic optimal transport. arXiv preprint arXiv:2006.16548, 2020

arXiv 2006

[3] [3]

A unified framework for hard and soft clustering with regularized optimal transport, 2024

Jean-Frédéric Diebold, Nicolas Papadakis, Arnaud Dessein, and Charles-Alban Deledalle. A unified framework for hard and soft clustering with regularized optimal transport, 2024. URL https://arxiv.org/abs/1711.04366

arXiv 2024

[4] [4]

Computational optimal transport: With applications to data science

Gabriel Peyr \'e , Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11 0 (5-6): 0 355--607, 2019

2019

[5] [5]

Sinkhorn distances: Lightspeed computation of optimal transport

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Neural Information Processing Systems (NeurIPS), 26, 2013

work page 2013

[6] [6]

Machine learning: a probabilistic perspective

Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012

2012

[7] [7]

Courty, and Valentin Emyia

Remi Flamary, Cédric Févotte, N. Courty, and Valentin Emyia. Optimal spectral transportation with application to music transcription. In Neural Information Processing Systems (NeurIPS), 2016

work page 2016