Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

Guoli Song; Qingming Huang; Qi Tian; Shuhui Wang

REVIEW 4 major objections 6 minor 68 references

Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that aligning each modality's Gaussian process kernel with the shared latent similarity improves cross-modal retrieval in GPLVM-based models.

desk verdict A modest, honest extension of the authors' own harmonization idea, but the trace variant is not a KL divergence as claimed, and the missing error bars undermine the empirical story. read the letter →

arxiv 1908.04979 v1 pith:QLBVN7NP submitted 2019-08-14 cs.LG cs.CVcs.IRstat.ML

Guoli Song , Shuhui Wang , Qingming Huang , Qi Tian This is my paper · ORCID

classification cs.LGcs.CVcs.IRstat.ML

keywords multimodallearningGaussianprocesslatentvariablemodelcross-modalretrievalkernelalignmentsimilaritymatrixharmonizationpriorGPLVMpreservation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multimodal Gaussian process latent variable models learn better shared representations when the training objective forces each modality's GP kernel to agree with the similarity structure of the latent points. The authors call this joint objective harmonization, and they instantiate it with three divergence penalties: the squared Frobenius norm, the $\ell_{2,1}$-norm, and a trace term that approximates the KL divergence between zero-mean Gaussians. Across PASCAL, Wiki, TVGraz, and MSCOCO, the harmonized versions of three GPLVM baselines report higher mean average precision for image-to-text and text-to-image retrieval than the corresponding non-harmonized models. If the claim holds, a lightweight additive regularizer can make non-parametric latent space models exploit cross-modal complementarity without changing their generative structure.

What carries the argument

The central object is the harmonization function $H_c(K_c,S_x)$, a convex penalty on the divergence between the modality-specific GP covariance matrix $K_c$ and the latent similarity matrix $S_x$ built from pairwise distances of latent points. Three instantiations are proposed: the squared Frobenius norm $\|K_c-S_x\|_F^2$, the $\ell_{2,1}$ norm $\|K_c-S_x\|_{2,1}$, and the trace penalty $\frac{1}{2}\operatorname{tr}(K_c^{-1}S_x)$, which the paper relates to the KL divergence between $\mathcal{N}(0,K_c)$ and $\mathcal{N}(0,S_x)$. The function enters as a factorized prior over $X$ and $\theta_c$, so the two modalities' GP mappings are no longer conditionally independent given $X$; the penalty is what transfers structural information between the observation, functional, and latent spaces.

What would settle it

Vary only the bandwidth $\gamma_x$ used to build the latent similarity matrix while keeping everything else fixed: if the reported retrieval gains disappear or reverse when the scale of $S_x$ moves away from the scale of $K_c$, the harmonization is exploiting scale compatibility rather than structural alignment.

Watch

Extended reading notes

Core claim

In the standard multimodal GPLVM, the latent coordinates $X$ and the kernel hyperparameters $\theta_c$ are learned under independent priors; this paper argues that this independence discards the mutual information between the functional mapping and the latent representation. The proposed harmonization prior factorizes as $p(\theta_1,\theta_2,X) \propto p(\theta_1,X)p(\theta_2,X)$, with each factor $p(\theta_c,X) = \exp(-\frac{1}{\sigma_c^2} H_c(K_c,S_x))$, so the objective becomes the sum of the usual negative log-likelihoods plus $\mu_c H_c(K_c,S_x)$. The paper claims that minimizing this objective with any of three convex harmonization functions pulls the modality-specific covariance matrices $K_1,K_2$ toward a common latent similarity $S_x$, which makes the two modalities share structure at the functional level and yields latent coordinates that are more semantically consistent. On the four datasets the harmonized variants are reported to achieve higher mAP than mGPLVM, m-SimGP, and m-RSimGP, with the trace form performing best in most settings.

Load-bearing premise

The method assumes that the kernel matrices and the latent similarity matrix are measured on comparable scales; if they are not, the penalty shrinks or distorts the kernels instead of aligning their structure.

Editorial extensions

If this is right

Across PASCAL, Wiki, TVGraz, and MSCOCO, the harmonized variants of mGPLVM, m-SimGP, and m-RSimGP all report higher mean average precision than their non-harmonized counterparts for both retrieval directions.
The trace harmonization function outperforms the Frobenius and $\ell_{2,1}$ forms in most configurations, indicating that ratio-based alignment of covariance matrices is the more effective of the three penalties.
Harmonization composes with existing latent-space semantic priors: adding it to m-RSimGP still improves retrieval over the same model without the harmonization term.
Training cost remains comparable to the GPLVM baselines, and the resulting latent embeddings are more compact and more semantically grouped, as shown by the paper's low-dimensional projections.
The sensitivity experiments show that a very large tradeoff weight $\mu$ degrades performance, so the harmonization term acts as a regularizer that needs moderate weighting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not explore it, but replacing the trace penalty with the full KL divergence, including the omitted log-determinant term, should make harmonization more robust to scale mismatch; if the paper's mechanism is right, retrieval should be preserved or improved across bandwidth choices.
Because the objective is additive in modalities, a direct extension to three or more modalities is to sum $H_c(K_c,S_x)$ over all $c$; the paper's claim predicts that each added modality tightens the common latent space and improves retrieval.
If the effect is structural rather than label-driven, the same harmonization should help in fully unsupervised multimodal settings and in missing-modality cases where one modality's GP kernel has to pull the latent space alone.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First things first: this is a modest but honest extension of the authors' ICCV 2017 harmonization work. What's new here is the l2,1 and trace variants of the regularizer, plus application to m-SimGP and m-RSimGP. On four benchmarks the harmonized variants consistently beat their non-harmonized baselines, though the gains are small on MSCOCO. The paper ships code, does parameter sensitivity analysis, and has a reasonable set of baselines. That deserves credit.

The soft spots are where the stress-test note lands. The paper claims the trace harmonization H = 1/2 tr(K^{-1}S) is 'related to the KL divergence' and forces the two Gaussian distributions to be close. That is not right. The actual KL has additional log-determinant terms, and without them the objective is minimized by inflating K, not by aligning it with S. The GP log-likelihood's log|K| term prevents blow-up, but the resulting optimum is a scale-balancing act, not structural alignment. So the stated mechanism for the best-performing variant is not what the objective actually enforces. The authors should either fix the theory, add the missing terms, or reframe the regularizer as a heuristic that works in practice. The same scale-incompatibility worry applies to the F-norm and l2,1 variants, since gamma_x is fixed to 1 while lengthscales are free.

The other weakness is experimental reporting. There are no error bars or significance tests, and the validation process is one vague sentence. For a paper whose main evidence is mAP differences of a few points, that's a real omission. The baselines m-SimGP and m-RSimGP are the authors' own models, which is fine, but it does mean the improvement is measured against the same group's previous work.

None of this destroys the paper's empirical case. The consistent gains across four datasets suggest the regularizer does something useful. But the theoretical framing is overreach, and the missing error bars make it hard to know how much of the improvement is robust.

This is a paper for the GPLVM-multimodal crowd, not a broad audience. I'd send it to a serious referee, but with a clear request to correct the KL claim and add variance information. It can become a solid short paper after revision.

Referee Report

4 major / 6 minor

Summary. The paper proposes a harmonization mechanism for multimodal Gaussian process latent variable models (GPLVMs). The authors define a joint prior over latent inputs X and kernel hyperparameters θ, implemented as a regularizer H_c(K_c, S_x) that penalizes disagreement between the modality-specific GP kernel K_c and an RBF similarity matrix S_x computed on the latent points. Three harmonization functions are proposed: the squared Frobenius norm, the l2,1 norm, and the trace of K_c^{-1}S_x. The regularizer is added to three base models, mGPLVM, m-SimGP, and m-RSimGP, yielding nine variants. Experiments on PASCAL, Wiki, TVGraz, and MSCOCO report consistent mAP improvements of the harmonized variants over their non-harmonized counterparts, along with parameter sensitivity analyses, latent-space visualizations, and diagnostics measuring the divergence between K_c and S_x.

Significance. If the proposed mechanism works as stated, the paper offers a simple and general way to couple latent positions with kernel hyperparameters in multimodal GPLVMs, and the breadth of the evaluation across three base models and four datasets is attractive. The manuscript includes publicly available code, a clear factored-prior formulation, and three different regularization geometries, which are useful assets. The main reservation is that the best-performing trace variant is not actually a divergence between N(0,K_c) and N(0,S_x), so the theoretical interpretation of the empirical gains is not supported as written. The empirical evaluation also lacks statistical significance testing, making the magnitude of the reported improvements uncertain.

major comments (4)

[Section 4.2, Eq. (14)] The claim that H_c = 1/2 tr(K_c^{-1}S_x) is 'related to the KL-divergence' and 'forces the two distributions to be close' is not correct. The KL divergence between N(0,K_c) and N(0,S_x) is 1/2(tr(K_c^{-1}S_x) - N + log|K_c| - log|S_x|), and the omitted log-determinant terms are exactly what make the divergence vanish when K_c = S_x. For the one-parameter family K_c = a S_x, the proposed H_c equals N/(2a), which decreases monotonically as a grows; the objective therefore rewards inflating the kernel relative to S_x. In the full objective (15) the likelihood's log|K_c| term prevents unbounded inflation, but the resulting balance point depends on d_c and μ, not on structural agreement. Since the trace variants achieve the best mAP in most of Tables 1-3, the empirical gains cannot be attributed to the stated KL/agreement mechanism unless the authors either use the true KL divergence or explicitly revise this claim.
[Section 6.6, Table 4] The paper's own diagnostic contradicts the central mechanism for the trace variant. Table 4 shows that for hmGPLVM the trace harmonization produces a total Riemannian distance of 38.7391, which is larger than the F-norm variant's 34.2681, while Table 1 shows that hmGPLVM(tr) has higher mAP (0.1732 vs 0.1613). Thus the trace variant's retrieval improvement is not explained by a smaller divergence between K_c and S_x; some other effect of the regularizer must be driving the gain. Please report the relationship between the harmonization distance and mAP, and provide evidence that the improved performance is due to alignment rather than, for example, scale shrinkage of the kernels.
[Section 6.3 and 6.3.2] All mAP numbers are reported from a single random train/test split, with no error bars, no multiple runs, and no significance tests. The phrase 'significant improvements' in Section 6.3.2 is therefore unsupported. Given that several reported gaps are only a few mAP points (e.g., hm-SimGP(F) vs m-SimGP on TVGraz in Table 2), the central empirical claim would be much stronger with mean ± std across at least five random splits and a paired significance test between each harmonized variant and its baseline.
[Section 4.1 and 6.2] The harmonization objectives implicitly assume scale compatibility between K_c and S_x, but this is not established. The latent similarity is fixed with γ_x = 1, while the GP kernels use free RBF lengthscales and variances. The F-norm and l2,1 penalties minimize absolute element-wise differences, so if the scales of K_c and S_x differ, the regularizer can be minimized by shrinking K_c rather than by aligning its correlation structure. The trace term is also scale-sensitive, as shown by the one-parameter example above. To support the structural interpretation, the authors should either normalize K_c and S_x (e.g., use correlation matrices) or include a scale parameter in S_x and report its learned value for each dataset.

minor comments (6)

[Section 4.2] The text says that any convex and sub-differentiable function can be used to define H_c, but after composition with the nonlinear kernel function k_c(X, θ_c), the overall objective is not convex in X and θ; please clarify that convexity is meant in the matrix argument only.
[Section 3.1, Eq. (5)] The m-SimGP likelihood uses S_c(S_c)^T, which for the symmetric similarity matrix S_c equals S_c^2, not S_c itself; the notation should be clarified to avoid confusion with the trace harmonization term tr(K_c^{-1}S_x).
[Table 4] The entry 'hm-SimGP (F' is missing a closing parenthesis.
[Section 6.3] The heading contains a typo: 'Perfomance' should be 'Performance'.
[Section 6.2] The description of the parameter validation process is vague; please specify the validation split and the range of hyperparameters searched for μ, λ, and the latent dimension.
[Figure 10] The color scales differ across panels, which makes visual comparison of absolute differences between K_c and S_x misleading; using a common color scale would strengthen the qualitative claim.

Circularity Check

1 steps flagged · score 2.0 of 10

Post-hoc verification of the harmonization objective is self-referential, but the central retrieval claim is evaluated on external labels and is not circular.

self definitional [Section 6.6 ('Analysis of the harmonization mechanism'), Table 4 and Fig. 10]
"In order to guarantee that the divergence between similarity structures in observed data space and the shared latent space to be small, we propose the harmonization constraints in Eq. (9) to preserve the structure consistency among GP kernels (K1,K2) and the latent similarity (Sx). ... Still the quantitative results demonstrate that the distance between GP kernels and the latent similarity is reduced in our models with the harmonization mechanism."

The harmonization objective in Eq. (11) is defined as minimizing Hc(Kc,Sx), with Hc given in Eqs. (12)-(14) as a distance or divergence between Kc and Sx. Table 4 and Fig. 10 verify that, after training, a related distance (Riemannian distance or absolute element-wise difference) between Kc and Sx is smaller for harmonized models. This is an optimization sanity check, not an independent confirmation: whenever the regularizer is active, the optimizer directly penalizes that quantity, so the observed reduction is true by construction. The central mAP claim is not circular because retrieval is scored on external class labels that are not used as the harmonization objective, so this self-referential analysis is not load-bearing for the paper's main result.

full rationale

The paper's central claim is that adding the harmonization regularizer Hc(Kc,Sx) improves cross-modal retrieval, and that claim is evaluated by mAP on held-out data with external class labels. Those labels do not define the harmonization objective, so the main empirical result is independent of the regularizer's definition. The only self-referential element is the mechanism analysis in Section 6.6, which shows that harmonized models reduce the gap between the GP kernels Kc and the latent similarity Sx; since Eqs. (11)-(14) define that gap as the training penalty, the reduction is expected by construction and is a sanity check rather than a prediction. The self-citations to the authors' prior m-SimGP and m-RSimGP papers are ordinary baseline construction and are not load-bearing. The trace harmonization's asserted KL-divergence connection is mathematically incomplete (it omits the log-determinant terms that would make the divergence vanish at Kc = Sx), but that is a correctness concern, not a circularity. Overall, no core derivation reduces to its own inputs, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim depends on a small number of tuned tradeoff parameters (mu, lambda, latent dimension), a fixed similarity bandwidth, and the core modeling assumption that aligning kernels with a fixed RBF similarity matrix is a beneficial inductive bias. No new physical entities are introduced; the harmonization prior is a mathematical construct validated only internally.

free parameters (5)

mu (mu1=mu2) = not reported per dataset; sensitivity shown in Fig. 6-7
Tradeoff weight for harmonization terms in Eq. (15)-(17); tuned by validation, exact values omitted.
lambda (lambda1=lambda2) = not reported per dataset
Tradeoff weight for semantic similarity/dissimilarity terms in hm-RSimGP Eq. (17); tuned by validation.
latent dimension q = selected within [7,10]
Dimensionality of shared latent space; chosen per dataset via validation.
gamma_x = 1
Bandwidth of latent similarity matrix S_x; fixed by hand (Section 6.2).
number of inducing points M = 100
Sparsification active set size for all models (Section 6.2).

assumptions (5)

domain assumption The two modalities Y1 and Y2 are generated conditionally independently from a shared latent variable X via GP mappings (Eq. 1).
Standard mGPLVM generative model; the paper relies on it for the likelihood factorization.
ad hoc to paper The joint prior factorizes as p(theta1,theta2,X) proportional to p(theta1,X)p(theta2,X) with p(theta_c,X)=exp(-(1/sigma_c^2)H_c(K_c,S_x)) (Eqs. 7, 10).
Core modeling choice defining harmonization; no external justification that this factorization is appropriate.
domain assumption The latent similarity S_x, computed with Euclidean distance and gamma_x=1, is a good structural target for both modality kernels.
Chosen without data-dependent adaptation; central to the harmonization objective.
standard math The harmonization functions H_c are convex and sub-differentiable, enabling gradient-based optimization (Section 4.2).
Convexity claims are correct for the three chosen functions, but the overall objective remains non-convex.
domain assumption MAP estimation with gradient descent (SCG) finds a good local optimum.
Standard practice for GPLVMs; no guarantees provided.

invented entities (1)

Harmonization prior
purpose: Joint prior over latent positions and kernel hyperparameters that aligns modality-specific GP kernels with the latent similarity matrix
New regularizer; its usefulness is supported only by the paper's own experiments on four datasets, with no independent theoretical or external validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Harmonized Multimodal Learning with Gaussian Process Latent Variable Models." pith.science (2026). https://pith.science/paper/QLBVN7NP

@misc{pith2026190804979,
  author       = {Pith},
  title        = {Pith review of: Harmonized Multimodal Learning with Gaussian Process Latent Variable Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QLBVN7NP}},
  note         = {Machine review of arXiv:1908.04979}
}

read the original abstract

Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs) to represent multimodal data in a common space. Previous multimodal GPLVM extensions generally adopt individual learning schemes on latent representations and kernel hyperparameters, which ignore their intrinsic relationship. To exploit strong complementarity among different modalities and GPLVM components, we develop a novel learning scheme called Harmonization, where latent model parameters are jointly learned from each other. Beyond the correlation fitting or intra-modal structure preservation paradigms widely used in existing studies, the harmonization is derived in a model-driven manner to encourage the agreement between modality-specific GP kernels and the similarity of latent representations. We present a range of multimodal learning models by incorporating the harmonization mechanism into several representative GPLVM-based approaches. Experimental results on four benchmark datasets show that the proposed models outperform the strong baselines for cross-modal retrieval tasks, and that the harmonized multimodal learning method is superior in discovering semantically consistent latent representation.

Figures

Figures reproduced from arXiv: 1908.04979 by the authors.

**Figure 2.** (a) Multimodal GPLVM (mGPLVM): Independent priors are im [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The proposed multimodal GPLVMs with harmonization. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: The performance comparison of different methods for cross-modal retrieval based on precision-recall curve. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Examples of cross-modal retrieval on Wiki dataset for the proposed models with the trace harmonization. The top three retrieved results [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Variants of hmGPLVM and hm-SimGP: average mAP score of two cross-modal retrieval tasks as a function of the harmonization parameters [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Variants of hm-RSimGP: average mAP score of two cross-modal [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Visualization of the discovered latent representations for the [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 10.** Figure 10: Visualization of the absolute element-wise difference between modality-specific GP kernels and the similarity matrix in the latent space on [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 67 canonical work pages

[1]

Deep correlation for matching images and text,

F. Yan and K. Mikolajczyk, “Deep correlation for matching images and text,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2015, pp. 3441–3450

work page 2015
[2]

Multimodal deep learning,

J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in Proc. Int. Conf. Mach. Learn. , 2011, pp. 689–696

work page 2011
[3]

Multimodal machine learning: A survey and taxonomy,

T. Baltru ˇsaitis, C. Ahuja, and L.-P . Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE T rans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443, 2019

work page 2019
[4]

Audio-visual speech recognition using deep learning,

K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata, “Audio-visual speech recognition using deep learning,” Appl. Intell., vol. 42, no. 4, pp. 722–737, 2015

work page 2015
[5]

Show and tell: A neural image caption generator,

O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3156–3164

work page 2015
[6]

Aligning where to see and what to tell: Image captioning with region-based attention and scene-speciﬁc contexts,

K. Fu, J. Jin, R. Cui, F. Sha, and C. Zhang, “Aligning where to see and what to tell: Image captioning with region-based attention and scene-speciﬁc contexts,” IEEE T rans. Pattern Anal. Mach. Intell., vol. 39, no. 12, pp. 2321–2334, 2017

work page 2017
[7]

On the role of correlation and abstraction in cross-modal multimedia retrieval,

J. C. Pereira, E. Coviello, G. Doyle, N. Rasiwasia, G. R. G. Lanck- riet, R. Levy, and N. Vasconcelos, “On the role of correlation and abstraction in cross-modal multimedia retrieval,” IEEE T rans. Pattern Anal. Mach. Intell. , vol. 36, no. 3, pp. 521–535, 2014

work page 2014
[8]

Joint feature selection and subspace learning for cross-modal retrieval,

K. Wang, R. He, L. Wang, W. Wang, and T. Tan, “Joint feature selection and subspace learning for cross-modal retrieval,” IEEE T rans. Pattern Anal. Mach. Intell. , vol. 38, no. 10, pp. 2010–2023, 2016

work page 2010

Show all 68 references

[9]

Ccl: Cross-modal correla- tion learning with multigrained fusion by hierarchical network,

Y. Peng, J. Qi, X. Huang, and Y. Yuan, “Ccl: Cross-modal correla- tion learning with multigrained fusion by hierarchical network,” IEEE T rans. Multimedia, vol. 20, no. 2, pp. 405–420, 2018

2018
[10]

Probabilistic non-linear principal component analysis with gaussian process latent variable models,

N. D. Lawrence, “Probabilistic non-linear principal component analysis with gaussian process latent variable models,” J. Mach. Learn. Res., vol. 6, pp. 1783–1816, 2005

2005
[11]

Learning shared latent structure for image synthesis and robotic imitation,

A. P . Shon, K. Grochow, A. Hertzmann, and R. P . N. Rao, “Learning shared latent structure for image synthesis and robotic imitation,” in Proc. Adv. Neural Inf. Process. Syst. , 2005, pp. 1233–1240

2005
[12]

Manifold relevance determination,

A. C. Damianou, C. H. Ek, M. K. Titsias, and N. D. Lawrence, “Manifold relevance determination,” in Proc. Int. Conf. Mach. Learn., 2012, pp. 145–152

2012
[13]

Gaussian process kernels for pattern discovery and extrapolation,

A. G. Wilson and R. P . Adams, “Gaussian process kernels for pattern discovery and extrapolation,” in Proc. Int. Conf. Mach. Learn., 2013, pp. 1067–1075

2013
[14]

Variational inference for latent variables and uncertain inputs in gaussian processes,

A. C. Damianou, M. K. Titsias, and N. D. Lawrence, “Variational inference for latent variables and uncertain inputs in gaussian processes,” J. Mach. Learn. Res. , vol. 2, 2015

2015
[15]

Variational inference for mahalanobis distance metrics in gaussian process regression,

M. K. Titsias and M. L ´azaro-Gredilla, “Variational inference for mahalanobis distance metrics in gaussian process regression,” in Proc. Adv. Neural Inf. Process. Syst. , 2013, pp. 279–287

2013
[16]

Discriminative shared gaussian processes for multiview and view-invariant facial expres- sion recognition,

S. Eleftheriadis, O. Rudovic, and M. Pantic, “Discriminative shared gaussian processes for multiview and view-invariant facial expres- sion recognition,” IEEE T rans. Image Process. , vol. 24, no. 1, pp. 189–204, 2015

2015
[17]

Multimodal similarity gaussian process latent variable model,

G. Song, S. Wang, Q. Huang, and Q. Tian, “Multimodal similarity gaussian process latent variable model,” IEEE T rans. Image Process., vol. 26, no. 9, pp. 4168–4181, 2017

2017
[18]

Training and evaluating multimodal word embeddings with large-scale web annotated images,

J. Mao, J. Xu, K. Jing, and A. L. Yuille, “Training and evaluating multimodal word embeddings with large-scale web annotated images,” in Proc. Adv. Neural Inf. Process. Syst. , 2016, pp. 442–450

2016
[19]

Similarity gaussian process latent variable model for multi-modal data analysis,

G. Song, S. Wang, Q. Huang, and Q. Tian, “Similarity gaussian process latent variable model for multi-modal data analysis,” in Proc. IEEE Int. Conf. Comput. Vis. , 2015, pp. 4050–4058

2015
[20]

Multimodal gaussian process latent variable models with harmonization,

G. Song, S. Wang, Q. Huang, and Q. Tian, “Multimodal gaussian process latent variable models with harmonization,” in Proc. IEEE Int. Conf. Comput. Vis. , 2017, pp. 5039–5047

2017
[21]

GPﬂow: A Gaussian process library using TensorFlow,

A. G. d. G. Matthews, M. van der Wilk, T. Nickson, K. Fujii, A. Boukouvalas, P . Leon-Villagra, Z. Ghahramani, and J. Hens- man, “GPﬂow: A Gaussian process library using TensorFlow,” J. Mach. Learn. Res. , vol. 18, no. 40, pp. 1–6, apr 2017

2017
[22]

Relations between two sets of variates,

H. Hotelling, “Relations between two sets of variates,” Biometrika, vol. 28, no. 3/4, pp. 321–377, 1936

1936
[23]

Canonical correlation analysis: An overview with application to learning methods,

D. R. Hardoon, S. Szedm ´ak, and J. Shawe-Taylor, “Canonical correlation analysis: An overview with application to learning methods,” Neural Comput., vol. 16, no. 12, pp. 2639–2664, 2004

2004
[24]

A new approach to cross-modal multimedia retrieval,

N. Rasiwasia, J. C. Pereira, E. Coviello, G. Doyle, G. R. G. Lanck- riet, R. Levy, and N. Vasconcelos, “A new approach to cross-modal multimedia retrieval,” in Proc. ACM Int. Conf. Multimedia, 2010, pp. 251–260

2010
[25]

Generalized multiview analysis: A discriminative latent space,

A. Sharma, A. Kumar, H. D. III, and D. W. Jacobs, “Generalized multiview analysis: A discriminative latent space,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2012, pp. 2160–2167

2012
[26]

Factorized orthogonal latent spaces,

M. Salzmann, C. H. Ek, R. Urtasun, and T. Darrell, “Factorized orthogonal latent spaces,” in Proc. Int. Conf. Artif. Intell. & Statistics, 2010, pp. 701–708

2010
[27]

Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora,

R. Socher and F. Li, “Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2010, pp. 966–973

2010
[28]

Learning the relative importance of objects from tagged images for retrieval and cross-modal search,

S. J. Hwang and K. Grauman, “Learning the relative importance of objects from tagged images for retrieval and cross-modal search,” Int. J. Comput. Vis. , vol. 100, no. 2, pp. 134–153, 2012

2012
[29]

On deep multi- view representation learning,

W. Wang, R. Arora, K. Livescu, and J. A. Bilmes, “On deep multi- view representation learning,” in Proc. Int. Conf. Mach. Learn., 2015, pp. 1083–1092

2015
[30]

Deep multimodal representation learning from temporal data,

X. Yang, P . Ramesh, R. Chitta, S. Madhvanath, E. A. Bernal, and J. Luo, “Deep multimodal representation learning from temporal data,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2017, pp. 5066–5074

2017
[31]

Joint global and co-attentive representation learning for image-sentence retrieval,

S. Wang, Y. Chen, J. Zhuo, Q. Huang, and Q. Tian, “Joint global and co-attentive representation learning for image-sentence retrieval,” in ACM Multimedia, Seoul, Republic of Korea, October 22-26, 2018 , 2018, pp. 1398–1406

2018
[32]

Deep canonical correlation analysis,

G. Andrew, R. Arora, J. A. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in Proc. Int. Conf. Mach. Learn. , 2013, pp. 1247–1255

2013
[33]

Overview and recent advances in par- tial least squares,

R. Rosipal and N. Kr ¨amer, “Overview and recent advances in par- tial least squares,” in Subspace, latent structure and feature selection , 2005, pp. 34–51. 14

2005
[34]

Cross-modal retrieval by real label partial least squares,

J. He, B. Ma, S. Wang, Y. Liu, and Q. Huang, “Cross-modal retrieval by real label partial least squares,” in Proc. ACM Int. Conf. Multimedia, 2016, pp. 227–231

2016
[35]

A probabilistic model for multimodal hash function learning,

Y. Zhen and D. Yeung, “A probabilistic model for multimodal hash function learning,” in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining , 2012, pp. 940–948

2012
[36]

Composite correlation quantization for efﬁcient multimodal retrieval,

M. Long, Y. Cao, J. Wang, and P . S. Yu, “Composite correlation quantization for efﬁcient multimodal retrieval,” in Proc. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval , 2016, pp. 579–588

2016
[37]

Deep cross-modal hashing,

Q. Jiang and W. Li, “Deep cross-modal hashing,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2017, pp. 3270–3278

2017
[38]

Learning multi-view neigh- borhood preserving projections,

N. Quadrianto and C. H. Lampert, “Learning multi-view neigh- borhood preserving projections,” in Proc. Int. Conf. Mach. Learn. , 2011, pp. 425–432

2011
[39]

Heterogeneous metric learning with joint graph regularization for cross-media retrieval,

X. Zhai, Y. Peng, and J. Xiao, “Heterogeneous metric learning with joint graph regularization for cross-media retrieval,” in Proc. AAAI Conf. Artif. Intell. , 2013

2013
[40]

Multi-feature metric learning with knowledge transfer among semantics and social tagging,

S. Wang, S. Jiang, Q. Huang, and Q. Tian, “Multi-feature metric learning with knowledge transfer among semantics and social tagging,” in CVPR, Providence, RI, USA, June 16-21 , 2012, pp. 2240– 2247

2012
[41]

Supervised coupled dictionary learning with group structures for multi-modal retrieval,

Y. Zhuang, Y. Wang, F. Wu, Y. Zhang, and W. Lu, “Supervised coupled dictionary learning with group structures for multi-modal retrieval,” in Proc. AAAI Conf. Artif. Intell. , 2013

2013
[42]

Generalized coupled dictionary learn- ing approach with applications to cross-modal matching,

D. Mandal and S. Biswas, “Generalized coupled dictionary learn- ing approach with applications to cross-modal matching,” IEEE T rans. Image Process., vol. 25, no. 8, pp. 3826–3837, 2016

2016
[43]

Modeling annotated data,

D. M. Blei and M. I. Jordan, “Modeling annotated data,” in Proc. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval , 2003, pp. 127–134

2003
[44]

Learning cross-modality similarity for multinomial data,

Y. Jia, M. Salzmann, and T. Darrell, “Learning cross-modality similarity for multinomial data,” in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 2407–2414

2011
[45]

Multimodal learning with deep boltzmann machines,

N. Srivastava and R. Salakhutdinov, “Multimodal learning with deep boltzmann machines,” J. Mach. Learn. Res. , vol. 15, no. 1, pp. 2949–2980, 2014

2014
[46]

Ambiguity modeling in latent spaces,

C. H. Ek, J. Rihan, P . H. S. Torr, G. Rogez, and N. D. Lawrence, “Ambiguity modeling in latent spaces,” in Machine Learning for Multimodal Interaction, 2008, pp. 62–73

2008
[47]

Deep gaussian processes,

A. C. Damianou and N. D. Lawrence, “Deep gaussian processes,” in Proc. Int. Conf. Artif. Intell. & Statistics , 2013, pp. 207–215

2013
[48]

Variational auto-encoded deep Gaussian processes,

Z. Dai, A. Damianou, J. Gonz ´alez, and N. Lawrence, “Variational auto-encoded deep Gaussian processes,” Proc. Int. Conf. Learn. Represent., 2016

2016
[49]

Local distance preservation in the GP-LVM through back constraints,

N. D. Lawrence and J. Q. Candela, “Local distance preservation in the GP-LVM through back constraints,” in Proc. Int. Conf. Mach. Learn., 2006, pp. 513–520

2006
[50]

Shared autoencoder gaussian process latent variable model for visual classiﬁcation,

J. Li, B. Zhang, and D. Zhang, “Shared autoencoder gaussian process latent variable model for visual classiﬁcation,” IEEE T rans. Neural Netw. Learn. Syst. , 2017

2017
[51]

Discriminative gaussian process latent variable models for classiﬁcation,

R. Urtasun and T. Darrell, “Discriminative gaussian process latent variable models for classiﬁcation,” in Proc. Int. Conf. Mach. Learn. , 2007

2007
[52]

Topologically-constrained latent variable models,

R. Urtasun, D. J. Fleet, A. Geiger, J. Popovi ´c, T. J. Darrell, and N. D. Lawrence, “Topologically-constrained latent variable models,” in Proc. Int. Conf. Mach. Learn. , 2008, pp. 1080–1087

2008
[53]

Multi-modal distance metric learning,

P . Xie and E. P . Xing, “Multi-modal distance metric learning,” in Proc. Int. Joint Conf. Artif. Intell. , 2013

2013
[54]

A scaled conjugate gradient algorithm for fast supervised learning,

M. F. Møller, “A scaled conjugate gradient algorithm for fast supervised learning,” Neural Networks , vol. 6, no. 4, pp. 525–533, 1993

1993
[55]

Col- lecting image annotations using amazon’s mechanical turk,

C. Rashtchian, P . Young, M. Hodosh, and J. Hockenmaier, “Col- lecting image annotations using amazon’s mechanical turk,” in NAACL HLT 2010 Workshop , 2010, pp. 139–147

2010
[56]

On the regularization of image semantics by modal expansion,

J. C. Pereira and N. Vasconcelos, “On the regularization of image semantics by modal expansion,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 3093–3099

2012
[57]

Tvgraz: Multi-modal learning of object categories by combining textual and visual features,

I. Khan, A. Saffari, and H. Bischof, “Tvgraz: Multi-modal learning of object categories by combining textual and visual features,” in AAPR Workshop, 2009, pp. 213–224

2009
[58]

Microsoft coco: Common objects in context,

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P . Perona, D. Ramanan, P . Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Proc. IEEE Eur. Conf. Comput. Vis. Springer, 2014, pp. 740–755

2014
[59]

Inception- v4, inception-resnet and the impact of residual connections on learning,

C. Szegedy, S. Ioffe, V . Vanhoucke, and A. A. Alemi, “Inception- v4, inception-resnet and the impact of residual connections on learning,” in Proc. AAAI Conf. Artif. Intell. , 2017, pp. 4278–4284

2017
[60]

C. D. Manning, P . Raghavan, and H. Sch ¨utze, Introduction to Information Retrieval. Cambridge University Press, 2008

2008
[61]

Learning consistent feature representation for cross-modal multimedia retrieval,

C. Kang, S. Xiang, S. Liao, C. Xu, and C. Pan, “Learning consistent feature representation for cross-modal multimedia retrieval,”IEEE T rans. Multimedia, vol. 17, no. 3, pp. 370–381, 2015

2015
[62]

Cross-modality binary code learning via fusion similarity hashing,

H. Liu, R. Ji, Y. Wu, F. Huang, and B. Zhang, “Cross-modality binary code learning via fusion similarity hashing,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2017, pp. 6345–6353

2017
[63]

A multi-view embed- ding space for modeling internet images, tags, and their seman- tics,

Y. Gong, Q. Ke, M. Isard, and S. Lazebnik, “A multi-view embed- ding space for modeling internet images, tags, and their seman- tics,” Int. J. Comput. Vis. , vol. 106, no. 2, pp. 210–233, 2014

2014
[64]

Multi-label cross- modal retrieval,

V . Ranjan, N. Rasiwasia, and C. V . Jawahar, “Multi-label cross- modal retrieval,” in Proc. IEEE Int. Conf. Comput. Vis. , 2015, pp. 4094–4102

2015
[65]

Visualizing data using t-sne,

L. Van der Maaten and G. Hinton, “Visualizing data using t-sne,” J. Mach. Learn. Res. , vol. 9, no. 2579-2605, p. 85, 2008

2008
[66]

A metric for covariance matrices,

W. F ¨orstner and B. Moonen, “A metric for covariance matrices,” in Geodesy-The Challenge of the 3rd Millennium . Springer, 2003, pp. 299–309

2003
[67]

Multimodal convolutional neural networks for matching image and sentence,

L. Ma, Z. Lu, L. Shang, and H. Li, “Multimodal convolutional neural networks for matching image and sentence,” in Proc. IEEE Int. Conf. Comput. Vis. , 2015, pp. 2623–2631. Guoli Song received the Ph.D. degree in com- puter engineering from University of Chinese Academy of Scie...

2015
[2012]

He is also with the Key Laboratory of Intelligent Information Pro- cessing, Chinese Academy of Sciences

He is currently an Associate Professor with the Institute of Computing Technology, Chi- nese Academy of Sciences. He is also with the Key Laboratory of Intelligent Information Pro- cessing, Chinese Academy of Sciences. His re- search interests include semantic image anal- ysis...

1988

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Deep correlation for matching images and text,

F. Yan and K. Mikolajczyk, “Deep correlation for matching images and text,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2015, pp. 3441–3450

work page 2015

[2] [2]

Multimodal deep learning,

J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in Proc. Int. Conf. Mach. Learn. , 2011, pp. 689–696

work page 2011

[3] [3]

Multimodal machine learning: A survey and taxonomy,

T. Baltru ˇsaitis, C. Ahuja, and L.-P . Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE T rans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443, 2019

work page 2019

[4] [4]

Audio-visual speech recognition using deep learning,

K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata, “Audio-visual speech recognition using deep learning,” Appl. Intell., vol. 42, no. 4, pp. 722–737, 2015

work page 2015

[5] [5]

Show and tell: A neural image caption generator,

O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3156–3164

work page 2015

[6] [6]

Aligning where to see and what to tell: Image captioning with region-based attention and scene-speciﬁc contexts,

K. Fu, J. Jin, R. Cui, F. Sha, and C. Zhang, “Aligning where to see and what to tell: Image captioning with region-based attention and scene-speciﬁc contexts,” IEEE T rans. Pattern Anal. Mach. Intell., vol. 39, no. 12, pp. 2321–2334, 2017

work page 2017

[7] [7]

On the role of correlation and abstraction in cross-modal multimedia retrieval,

J. C. Pereira, E. Coviello, G. Doyle, N. Rasiwasia, G. R. G. Lanck- riet, R. Levy, and N. Vasconcelos, “On the role of correlation and abstraction in cross-modal multimedia retrieval,” IEEE T rans. Pattern Anal. Mach. Intell. , vol. 36, no. 3, pp. 521–535, 2014

work page 2014

[8] [8]

Joint feature selection and subspace learning for cross-modal retrieval,

K. Wang, R. He, L. Wang, W. Wang, and T. Tan, “Joint feature selection and subspace learning for cross-modal retrieval,” IEEE T rans. Pattern Anal. Mach. Intell. , vol. 38, no. 10, pp. 2010–2023, 2016

work page 2010