Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Kernelized Gromov–Wasserstein transport aligns several modalities onto a user-chosen latent distribution, converges at the parametric $n^{-1/2}$ sampling rate for its empirical objective, and improves MS-COCO image–text retrieval over…

desk verdict Solid OT theory with a load-bearing PSD-kernel assumption that the experiments neither verify nor satisfy, plus an overclaimed purity result. read the letter →

arxiv 2608.04234 v1 pith:TUAFA7BM submitted 2026-08-04 math.ST cs.LGstat.MLstat.TH

Yixuan Florence Wu , Yilun Zhu , Naichen Shi This is my paper · ORCID
classification math.STcs.LGstat.MLstat.TH MSC 49Q2262G2046E22
keywords multimodalalignmentkernelGromov–Wassersteinoptimaltransportentropicfinite-rankMercerapproximationsamplecomplexityvariationalliftingrepresentationlearningPivotedCholesky
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that several frozen single-modality encoders can be aligned into one shared latent space, with the geometry and marginal distribution of that space prescribed by the user, even when cross-modal paired data are scarce. It introduces a joint kernel entropic Gromov–Wasserstein (JK-EGW) objective, a transport framework that aligns distributions through their pairwise relational structure rather than raw coordinates, in which a global affinity kernel scores pairs of observations both within and across modalities and each modality is coupled by an optimal transport plan to a common reference measure $\nu$. The core theoretical claim is that despite the nonconvexity of the quadratic program, the empirical objective converges to its population counterpart at the parametric rate $n^{-1/2}$, plus a kernel-truncation term that decays with the Mercer tail of the kernel. The core practical claim is that the objective is solvable by an alternating scheme whose transport-plan step is a batch of entropic optimal transport problems, and that the resulting shared representations improve bidirectional image–text retrieval on MS-COCO over three recent baselines. A sympathetic reader would care because scarce paired data is the practical bottleneck in multimodal learning, and this result promises a provably convergent, computationally tractable way to control the geometry of the shared space without training a heavy joint encoder.

What carries the argument

The load-bearing object is the variational lifting of the rank-truncated objective (Theorem 4), which converts the nonconvex quadratic interaction into a linear entropic optimal transport problem. For a fixed truncation rank $R$, the rank-$R$ JK-EGW objective equals the infimum over an auxiliary matrix $A \in \mathbb{R}^{R\times d_z}$ of $2\|A\|_F^2 + \sum_i \mathrm{OT}^i_{A,m_\Phi,\epsilon}(\mu_i,\nu)$, where each $\mathrm{OT}^i$ is a standard entropic transport problem with cost $c^i_{A,m_\Phi}(x,z) = 2\Phi_R(x)^\top m_\Phi \|z\|^2 - 4\langle A,\Phi_R(x)z^\top\rangle_F$; here $\Phi_R(x)$ stacks the first $R$ scaled Mercer eigenfunctions (the Mercer decomposition being the spectral expansion of a positive-semidefinite kernel into orthogonal eigenfunctions) and $m_\Phi = \sum_j \int \Phi_R \, d\mu_j$ is the averaged feature moment. The identity that does the work is $-2\|\sum_i V_i\|_F^2 = \inf_A (2\|A\|_F^2 - 4\langle A, \sum_i V_i\rangle_F)$ with $V_i = \int \Phi_R(x) z^\top d\pi_i$, which moves the quadratic dependence on the couplings into a bounded auxiliary variable $A$. This reduction carries both halves of the paper: statistically it lets the proof borrow linear-EOT sample-complexity and potential-regularity machinery, yielding the $n^{-1/2}$ rate without tail conditions on the modality measures; computationally it yields Algorithm 1, an alternating scheme in which the transport-plan step is a batch of Sinkhorn-style entropic solves and the $A$-step is the closed-form update $A = L^\top \hat\Pi \hat Z$ applied to the Pivoted-Cholesky factor $L$ (a greedy low-rank factorization) of the empirical kernel matrix.

What would settle it

On the exact MS-COCO configurations of Section 5.3, construct the full empirical block kernel matrix $W$ (within-modality similarity blocks and the cross-modality affinity blocks as actually defined for those experiments) and compute its smallest eigenvalue and its Mercer tail $\eta_R$ at the rank used by Algorithm 1; a negative eigenvalue of size comparable to the diagonal scale settles that Assumption 1 fails for the evaluated kernel, so Theorems 5 and 9 do not formally apply to the retrieval numbers, and a slow decay of $\eta_R$ would inflate the truncation term far beyond the claimed rate.

Watch

Extended reading notes

Core claim

The discovery is that a kernelized Gromov–Wasserstein alignment objective, though it is a nonconvex quadratic program over the couplings, becomes statistically and computationally tame once the global affinity kernel $K$ admits a finite-rank Mercer truncation. Writing the kernel as $K(x,x') = \sum_{r\ge 1} \lambda_r \varphi'_r(x)\varphi'_r(x')$ and keeping the first $R$ scaled eigenfunctions $\Phi_R(x)$, the quadratic interaction among couplings depends on each coupling only through the feature-latent moment $\int \Phi_R(x) z^\top d\pi_i$, so the rank-$R$ objective lifts (Theorem 4) to an infimum over an auxiliary matrix $A$ of $2\|A\|_F^2$ plus a sum of linear entropic optimal transport problems with cost $c^i_{A,m_\Phi}(x,z) = 2\Phi_R(x)^\top m_\Phi \|z\|^2 - 4\langle A, \Phi_R(x) z^\top\rangle_F$, where $m_\Phi$ is the averaged feature moment. That lifting routes the statistics through linear-EOT empirical-process bounds: Theorem 5 bounds the two-sample deviation $\mathbb{E}|S_\epsilon(\{\mu_i\},\nu) - S_\epsilon(\{\hat\mu_i^n\},\hat\nu_n)|$ by a constant times $n^{-1/2}$ plus $\sigma\eta_R$, with constants depending on the number of modalities, the latent dimension, the truncation rank, and the ratio $\sigma/\epsilon$ but not on the ambient feature dimension, and Theorem 9 extends the same parametric rate to the fully computable objective with a Pivoted-Cholesky low-rank kernel factor. Empirically, on a 6K and a 9K MS-COCO image–text benchmark with a 20-component Gaussian-mixture reference distribution in 32 latent dimensions, JK-EGW is reported as the only method among those compared with positive silhouette scores and as the best on bidirectional R@1 and R@5 retrieval, ahead of the CSA, CSA+STRUCTURE, and MLP+STRUCTURE baselines.

Load-bearing premise

Every theoretical guarantee in the paper rests on a single premise: the global affinity kernel that scores similarities within and across modalities is symmetric positive semidefinite, has a uniformly bounded diagonal, and has a Mercer tail $\eta_R = \sup_x \sum_{r>R} \lambda_r \varphi'_r(x)^2$ that vanishes as the truncation rank grows, and the experimental configurations with label-agreement cross-modality blocks are never checked against this premise.

Editorial extensions

If this is right

  • If the rank-$R$ kernel truncation has a fast-decaying spectrum (periodized RBF kernels give $\eta_R$ decaying faster than any polynomial in $R$), the effective statistical complexity of JK-EGW is governed by the latent dimension $d_z$ and the truncation rank $R$, not by the raw feature dimension of the modalities.
  • For fixed regularization $\epsilon$, the two-sample error decomposes into a statistical term of order $n^{-1/2}$ and a kernel-truncation term of order $\sigma\eta_R$; choosing $R$ so that $\eta_R \ll n^{-1/2}$ makes the empirical objective match the known parametric rate up to constants.
  • Because the user chooses the reference measure $\nu$, the practitioner can impose a target geometry on the shared space, such as the Gaussian mixture with means along orthogonal directions used in the experiments, and retain the same $n^{-1/2}$ guarantee.
  • The alternating algorithm costs $O(Mn^2)$ per iteration for the entropic transport updates, with the one-time Pivoted-Cholesky factorization costing $O(N\hat R^2)$ arithmetic operations and $O(N\hat R)$ memory for $N = Mn$, making the method applicable when $N$ is in the tens of thousands.
  • On the tested MS-COCO subsets, JK-EGW attains the highest purity, the only positive silhouette score, and the best R@1 and R@5 in both retrieval directions, with the retrieval advantage persisting across latent dimensions $d \in \{32, 48, 64, 128\}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A check the paper does not run: the MS-COCO experiments of Section 5.3 do not specify how the cross-modality affinity blocks are built, and the numerals experiment uses label agreement, which is not guaranteed to be positive semidefinite; computing the smallest eigenvalue of the empirical block kernel $W$ on those configurations would settle whether Assumption 1 covers the reported numbers.
  • Because the lifted objective is linear in the aggregate feature-latent moment $A$, the reference distribution $\nu$ could itself be optimized inside the same variational form, over a parametric family such as Gaussian mixtures with learnable means, turning the latent geometry from a fixed design choice into part of the alignment task; the authors list adapting $\nu$ as future work but do not pursu
  • The $n^{-1/2}$ bound is independent of the raw feature dimension, which makes JK-EGW a natural candidate for aligning very high-dimensional frozen encoder embeddings; whether that advantage materializes depends on the practical rank $\hat R$ of the empirical kernel, since the factorization cost scales with $\hat R$ rather than the ambient dimension.
  • The paper's validation of the rate (Section 5.2) carries its own stated caveat that optimization error is assumed negligible against statistical error, so a clean experimental confirmation of Theorem 5 would need a certified optimality gap rather than a converged objective value.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes JK-EGW, a joint kernel entropic Gromov--Wasserstein formulation for aligning multiple modalities into a shared latent space with a prescribed reference distribution. The objective uses a global affinity kernel on the disjoint union of modality spaces, combines within- and cross-modality similarities, and is regularized by KL entropy. The theoretical core assumes the kernel is positive semidefinite with a uniformly controlled Mercer tail (Assumption 1), then proves a kernel-truncation error bound (Proposition 3), a variational lifting of the rank-truncated problem to a family of entropic optimal transport problems (Theorem 4), and a parametric O(n^{-1/2}) sample complexity bound for the full-kernel objective (Theorem 5). The empirical part develops a low-rank Pivoted Cholesky approximation and an alternating Sinkhorn algorithm (Algorithm 1), with an end-to-end error theorem (Theorem 9). Experiments include a digit multi-feature example, a synthetic finite-support rate study, and MS-COCO image--text alignment where JK-EGW is compared with CSA and STRUCTURE baselines.

Significance. If the theoretical claims hold under Assumption 1, the paper gives a clean separation between statistical error and kernel-truncation error for a kernelized multi-modal optimal transport problem, and the variational lifting is an elegant way to convert a quadratic GW-type problem into solvable EOT subproblems. The n^{-1/2} rate, the explicit role of the latent dimension and Mercer rank rather than raw feature dimension, and the synthetic validation of the rate are genuine strengths. However, the paper's own experiments appear to use cross-modality affinity blocks that are not guaranteed to be positive semidefinite, which is exactly what Assumption 1 requires. Because the stated theory does not cover the reported experimental configurations, the empirical claims are not currently supported by the manuscript's theorems. This is a load-bearing gap that requires a major revision.

major comments (3)
  1. [Section 2.2 (Assumption 1) vs. Sections 5.1 and 5.3] Assumption 1 is load-bearing for Proposition 3, Theorem 4, Theorem 5, Corollary 7, Proposition 12, and the Pivoted Cholesky guarantee in Section 4.1. In Section 5.1 the cross-modality block is defined by Wi,j_{k,k'} = 1{same digit label}. Such a block matrix is not PSD in general: with n=2 per modality, taking the RBF within-block R=[[1,a],[a,1]] for 0<a<1 and the all-ones cross-block J, the block matrix [[R,J],[J,R]] has eigenvalue a-1<0 with eigenvector (1,1,-1,-1). Thus the Mercer decomposition and the positive-semidefinite residual property of Pivoted Cholesky do not apply to this configuration. In Section 5.3 the cross-modality kernel is not specified at all, and no minimum-eigenvalue check is reported. Consequently the reported retrieval improvements are not backed by the paper's theoretical guarantees. The authors should either construct and verify PSD empirical kernels (for example by projection onto the PSD cone) or clearly state that the experimental configurations lie outside the certified regime.
  2. [Section B.1, Proposition 6 proof] The proof of the rank-truncated sample complexity bound needs a uniform-in-A dual comparison. After Eq. (12), the text says 'By the same dual-comparison argument as in Corollary 2 of Mena and Niles-Weed (2019), uniformly over A in A_L, ...'. Lemma 13 provides regularity of optimal potentials for a fixed A, and the subsequent empirical-process bounds are uniform in the potential classes, but the manuscript does not prove that the dual comparison can be made simultaneously for all A in A_L. Since the supremum over A is essential for bounding the difference of the lifted objectives, this step is load-bearing. Please supply the missing uniform argument, or state and prove a uniform-in-A version of the potential regularity / dual comparison.
  3. [Section 4.1 and Proposition 12] The Pivoted Cholesky error analysis explicitly relies on W being positive semidefinite: 'Under the positive-semidefinite kernel assumption, the Pivoted Cholesky factorization ... yields a positive-semidefinite residual W-LL^T ⪰ 0 at each truncation level.' If the empirical block matrix is indefinite, this residual need not be PSD and the bound rho_Rhat <= tau_Rhat cannot be guaranteed. Theorem 9 then invokes rho_Rhat <= tau_Rhat almost surely, but Algorithm 1 does not verify this condition. At minimum, the paper should add a PSD check or modify the algorithm to handle indefinite empirical kernels, and then restate the corresponding error guarantees.
minor comments (5)
  1. [Section 5.3 and Appendix E, Tables 1 and 3] The sentence 'JK-EGW produces the strongest alignment and retrieval results across all dimensions' is not supported by Appendix E: on the 9K subset, the purity of JK-EGW is below CSA and CSA+STRUCTURE at several latent dimensions. Please qualify the claim to retrieval and geometric coherence, or discuss the purity trade-off.
  2. [Algorithm 1 and surrounding text] The text says 'the purple lines execute the sub-optimization for the transport plans and the maroon lines optimize A', but the printed algorithm contains no colored lines. Please remove the color references or add colors.
  3. [Section 3.1] The sentence 'Let x1,...,xM i.i.d.~rho gives a dataset of {x1_k,...,xM_k}' is confusing: if each modality has its own distribution mu_i, the samples should be i.i.d. from the appropriate marginal or from rho conditional on each modality component. Please clarify the sampling model.
  4. [Figure 4] The text in Section 5.2 reports n in {50,100,200,400,800,1600}, but the horizontal axis of Figure 4 is labeled 2^6 through 2^10, i.e. {64,128,256,512,1024}. Please make the sample sizes in the figure consistent with the text.
  5. [Abstract and author affiliations] There are several typos and grammatical issues: 'Deparment' in the affiliation, 'allowing us to take the advantage' in the abstract, and 'instantiates as' in Section 5.3. A copyedit would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the n^{-1/2} sample-complexity bound is derived from explicit Mercer-tail and sub-Weibull assumptions, not imposed or fitted.

full rationale

The paper's derivation chain is assumption-to-theorem rather than definition-to-definition. The objective S_epsilon in Eq. (1) is defined independently of the later theory; Assumption 1 then supplies the Mercer decomposition and the residual bound eta_R, under which Proposition 3 bounds the truncation error |S_epsilon - S^R_epsilon| by M^2 sigma eta_R using only Cauchy-Schwarz and the second moment of the latent measure. Theorem 4 is an algebraic variational lifting of the rank-R objective, with the auxiliary matrix A introduced through the standard identity -2||sum V_i||_F^2 = inf_A (2||A||_F^2 - 4 <A, sum V_i>), and no fitted quantity is relabeled as a prediction. The central claim, Theorem 5, is proved by a triangle decomposition into two truncation terms and a rank-R statistical term, the last being controlled in Proposition 6 by empirical-process bounds over EOT potentials whose regularity is established in Lemma 13; the eta_R term is then carried along explicitly, so the parametric 1/sqrt(n) rate is a proven consequence of the stated kernel tail and moment assumptions, not an imposed rate. The paper cites external results (Mercer's theorem, Steinwart and Scovel, Mena and Niles-Weed, Zhang et al.) for standard ingredients, and there are no load-bearing self-citations or imported uniqueness theorems. The synthetic experiment computes both the empirical and population objectives with the same solver and observes a curve consistent with the bound; this is a validation exercise, not a case where the predicted quantity is the fitted input. The only substantive concern is that Assumption 1 may not be verified for the MS-COCO label-agreement or unspecified cross-modality blocks, which is a correctness/validity risk and not a circularity, so under the hard rules it does not raise the circularity score. Overall, the derivation is self-contained given its assumptions, and no circular step can be quoted from the paper.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central theoretical result is derived under explicit regularity assumptions on K and ν; these are the axioms the reader must accept. The kernel PSD condition is the most fragile because the experiments do not verify it. The method itself has several user-chosen hyperparameters (ε, ν, K, ranks), which are not fitted to data in the theory but are free design choices.

free parameters (4)
  • Entropic regularization parameter ε = 0.2 (synthetic); not reported for MS-COCO
    Controls the convexity/entropy of the transport problem and enters the sample-complexity constants; a user-chosen hyperparameter, not fitted to data to produce the rate.
  • Latent reference distribution ν = GMM with 20 components, d=32, means along orthogonal directions (distance and covariance not specified)
    The method maps every modality onto this prescribed distribution; its shape is a design choice and is assumed 4-sub-Weibull in the theory.
  • Affinity kernel K and its bandwidth(s) = RBF ℓ=0.1/0.15, γ=1/0.6 in synthetic; unspecified for MS-COCO
    The global affinity kernel is the core input of the objective; its construction is data- and task-dependent and is not specified for the main experiment.
  • Truncation ranks R (Mercer) and R̂ (computational) = Pivoted Cholesky tolerance τ_R̂; R in theory is a free truncation level
    Approximation error depends on kernel tail; increasing ranks reduces η_R and ρ_R̂ but costs computation.
assumptions (4)
  • domain assumption K is symmetric positive semidefinite and satisfies the uniform diagonal bound and Mercer-tail control η_R → 0 (Assumption 1, Section 2.2)
    Necessary for the Mercer decomposition, the truncation error bound, the variational lifting, and the Pivoted Cholesky low-rank factorization; not verified in experiments.
  • domain assumption ν is 4-sub-Weibull with parameter σ² (Section 1.2)
    Controls tails of the latent reference distribution; used in EOT potential regularity, moment bounds, and the constants in Theorem 5.
  • domain assumption Modality samples x^i_k are drawn i.i.d. from μ_i, with z_l i.i.d. from ν, independent across modalities and from z (Section 3.1)
    Underlies the empirical process arguments and the n^{-1/2} rate; it ignores dependence across modalities in paired data.
  • standard math Mercer eigenfunctions are bounded by κ in L∞ (part of Assumption 1)
    Uniform diagonal bound K(x,x) ≤ κ² implies ∥Φ_R(x)∥ ≤ κ, used throughout the proofs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport." pith.science (2026). https://pith.science/paper/TUAFA7BM

@misc{pith2026260804234,
  author       = {Pith},
  title        = {Pith review of: Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TUAFA7BM}},
  note         = {Machine review of arXiv:2608.04234}
}
abstract

We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimodal encoders are available but cross-modal paired data are scarce. We propose a structure-preserving alignment framework, joint kernel entropic Gromov--Wasserstein Optimal Transport (JK-EGW), which maps multiple modalities into a common latent space by minimizing a quadratic optimal transport objective. JK-EGW leverages fine-grained similarity relationships within and across modalities to construct a global affinity kernel instead of relying on raw feature-space distances. Our framework naturally provides explicit control over the geometry and distribution of the latent embedding. On the theory side, we establish parametric sample complexity rate of $n^{-1/2}$, matching the corresponding rates for standard, entropic and Gromov--Wasserstein optimal transport. On the algorithmic side, we derive a scalable alternating procedure to solve JK-EGW with entropic optimal transport (EOT) updates through a low-rank kernel approximation and a variational lifting. This lifting scheme effectively relieves the burden of a quadratic objective, and allowing us to take the advantage of existing EOT solvers. Empirically, we focus on post-hoc alignment of embeddings from pretrained encoders in data-scarce regimes, and show that our proposed method achieves improved multimodal retrieval performance compared to existing alignment baselines.

Figures

Figures reproduced from arXiv: 2608.04234 by the authors.

Figure 1
Figure 1. A global affinity kernel K scores observation pairs within and across modalities. Each modality is coupled to a prescribed latent reference ν. problem. JK-EGW aligns the distribution of each modality with a common reference distribution in the latent space through probabilistic couplings. The objective encourages observations with high affinity to link to nearby latent representations, thereby unifying within-modali… view at source ↗
Figure 2
Figure 2. Worked illustration of Algorithm 1 on a multi-view handwritten digits dataset. Two feature views, Fourier coefficients (FT) and Karhunen–Lo`eve coefficients (KL), are treated as two modalities and coupled to a shared two-dimensional Gaussian latent reference. Points show the barycentric latent embeddings induced by the learned transport plans and are colored by digit class. 20 [PITH_FULL_IMAGE:figures/full_fig_p020… view at source ↗
Figure 3
Figure 3. The feature space follow two discrete mixtures of Gaussian distributions. The latent distribution is uniform on a 2D grid. Parametric rate. For each n ∈ {50, 100, 200, 400, 800, 1600}, we sample and construct empirical measures µˆ 1 n , µˆ 2 n , and νˆn, independently across modalities and the latent reference, and calculate Sϵ({µˆ i n}, νˆn) and Sϵ({µ i}, ν) via Algorithm 1. Since the population distributions µ 1 ,… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Left: Sample complexity of the full-kernel entropic objective. Mean absolute error of the plug-in estimator versus n (log–log), in the one-sample (ν known) and two-sample (ν estimated) regimes. Right: Objective truncation gap |Sϵ − S R ϵ | v.s. Mercer rank R. Objective…
Figure 5
Figure 5. Figure 5: Two-dimensional t-SNE embeddings of the shared representations learned by JK-EGW and the baselines on the MS–COCO 6K subset. 6 Conclusion We propose JK-EGW problem that maps several frozen unimodal encoders into a shared latent space with a prespecified marginal distri…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 8 canonical work pages

  1. [1]

    Jensen’s inequality then gives E∥Y 1∥ ≤ E∥Y 1∥2 1/2 , and summing overj∈[M] gives the display

    2 ≤ κ2 n, where the last inequality uses ∥ΦR(x)∥ ≤κ. Jensen’s inequality then gives E∥Y 1∥ ≤ E∥Y 1∥2 1/2 , and summing overj∈[M] gives the display. We also need to specify the radiusL. Define eσ:=σ∨inf s>0 : Z exp ∥z∥4 2s2 dˆνn(z)≤2 , then bothνand ˆν n are 4-sub-Weibull with parametereσ2. We now choose L=Mκ(2eσ 2)1/4. 37 Wu, Zhu, and Shi With this choice...

    work page 2022

  2. [2]

    Fixk∈N, define τ 2 k =σ 2∨ kσ2 n nX l=1 exp(∥zl∥4/(2kσ2)) ! so thatν,ˆνn are both 4-sub-Weibull with parameterτ 2 k

    It remains to bound moments of ˜σ. Fixk∈N, define τ 2 k =σ 2∨ kσ2 n nX l=1 exp(∥zl∥4/(2kσ2)) ! so thatν,ˆνn are both 4-sub-Weibull with parameterτ 2 k . By Lemma 4 of Mena and Niles-Weed (2019) and similar to Theorem 2 of Zhang et al. (2024), we have E[˜σ2k]≲ k σ2k. Hence, setq R,z = R∨dz 2 , we have E (1 + ˜σ4s) sup θ∈Fs |(µ−ˆµn)θ| ≤ s E[(1 + ˜σ4s)2] (κ+...

    work page 2019

  3. [5]

    With limited data for multimodal alignment, let the structure guide you.arXiv preprint arXiv:2506.16895,

    Fabian Gr¨ oger, Shuo Wen, Huyen Le, and Maria Brbi´ c. With limited data for multimodal alignment, let the structure guide you.arXiv preprint arXiv:2506.16895,

    work page arXiv

  4. [6]

    The np-hardness of the gromov-wasserstein distance.arXiv preprint arXiv:2408.06525,

    Natalia Kravtsova. The np-hardness of the gromov-wasserstein distance.arXiv preprint arXiv:2408.06525,

    arXiv

  5. [10]

    Ape: Aligning pretrained encoders to quickly learn aligned multimodal representations.arXiv preprint arXiv:2210.03927,

    Elan Rosenfeld, Preetum Nakkiran, Hadi Pouransari, Oncel Tuzel, and Fartash Faghri. Ape: Aligning pretrained encoders to quickly learn aligned multimodal representations.arXiv preprint arXiv:2210.03927,

    arXiv

  6. [11]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu

    URLhttps://arxiv.org/abs/2508.10104. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding.Advances in neural information processing systems, 33:16857–16867,

    arXiv

  7. [12]

    Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo

    URLhttps://openreview.net/forum?id=Z4R2rkPgBy. Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models.arXiv preprint arXiv:1611.01891,

    arXiv

  8. [14]

    Learning factorized multimodal representations.arXiv preprint arXiv:1806.06176,

    Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Rus- lan Salakhutdinov. Learning factorized multimodal representations.arXiv preprint arXiv:1806.06176,

    arXiv

Show all 20 references
  1. [16]

    30 JK-EGW for Multimodal Alignment Appendix A. Proofs for Section 2 A.1 Proof of Proposition 2 We first prove the claims for general kernels and specialize to the RBF kernel, then prove parts (a) and (b) by computing the Mercer eigensystem of a stationary kernel on [0 , 1]dx e...

    2012

  2. [18]

    (1 + ˜σs) sup ψ∈Gs |(ν−ˆνn)ψ| # ≤ vuutE[(1 + ˜σs)2]E

    |α|/2 , for every multi-index α with 1≤α , and some constant Cα> 0 that depends on α. Similarly, for anyσ, letG σ be the class ofC ∞(Rdz) functionsψ(z) satisfying: ψ(z)≤2Mκ 2∥z∥2 + 4·2 1/4Mκ 2σ1/2∥z∥+ 3 √ 2Mκ 2σ, −ψ(z)≤2Mκ 2∥z∥2 + 4·2 1/4Mκ 2σ1/2∥z∥+ 5 √ 2Mκ 2σ, |Dα zψ(z)|≤K α...

    2019

  3. [20]

    In addition,νis 4-sub-Weibull with parameterσ 2, so E h SR ϵ ({ˆµi n}M i=1,ˆνn)− ˆSϵ({ˆµi n}M i=1,ˆνn) i ≤M 2ηRσ

    Now, X k,k′ ˆπi k,ℓ ˆπj k′,ℓ′ = X k ˆπi k,ℓ ! X k′ ˆπj k′,ℓ′ ! = 1 n· 1 n. In addition,νis 4-sub-Weibull with parameterσ 2, so E h SR ϵ ({ˆµi n}M i=1,ˆνn)− ˆSϵ({ˆµi n}M i=1,ˆνn) i ≤M 2ηRσ. 42 JK-EGW for Multimodal Alignment C.3 Proof of Proposition 12 We follow the proof in Se...

    2022

  4. [1909]

    Relative representations enable zero-shot latent space communication

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodol` a. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430,

    arXiv

  5. [1984]

    Beats: Audio pre-training with acoustic tokenizers.arXiv preprint arXiv:2212.09058,

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers.arXiv preprint arXiv:2212.09058,

    arXiv

  6. [1996]

    Git: A generative image-to-text transformer for vision and language.arXiv preprint arXiv:2205.14100,

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language.arXiv preprint arXiv:2205.14100,

    arXiv

  7. [1998]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang

    DOI: https://doi.org/10.24432/C5HC70. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p...

    2023 doi

  8. [2016]

    Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    arXiv

  9. [2022]

    A statistical theory of contrastive pre-training and multimodal generative ai.arXiv preprint arXiv:2501.04641,

    Kazusato Oko, Licong Lin, Yuhang Cai, and Song Mei. A statistical theory of contrastive pre-training and multimodal generative ai.arXiv preprint arXiv:2501.04641,

  10. [2023]

    Vse++: Improving visual-semantic embeddings with hard negatives.arXiv preprint arXiv:1707.05612,

    Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives.arXiv preprint arXiv:1707.05612,

    arXiv

  11. [2024]

    25 Wu, Zhu, and Shi David Alvarez-Melis and Tommi Jaakkola

    URL https://openreview.net/forum?id=5jcav5RcKw. 25 Wu, Zhu, and Shi David Alvarez-Melis and Tommi Jaakkola. Gromov-wasserstein alignment of word embedding spaces. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 1881–1890,

    2018

  12. [2025]

    Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al

    URLhttps://openreview.net/forum?id=6Mg7pjG7Sw. Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal 27 Wu, Zhu, and Shi representation learning.Advances in ...

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.