Pith. sign in

REVIEW 3 major objections 6 minor 91 references

Face and voice features of the same person can be pulled into a shared convex hull by learning intermediate convex embeddings, cutting false matches on VoxCeleb.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 16:56 UTC pith:7BOXSLVM

load-bearing objection Clean empirical win on face–voice association via attention-weighted interpolants; the geometry talk is looser than the numbers. the 3 major comments →

arxiv 2607.28129 v1 pith:7BOXSLVM submitted 2026-07-30 cs.CV cs.IR

Face and Voice Cross-modal Association with Learning Convex Feature Embedding

Taewan Kim , Jiwoo Kang This is my paper
classification cs.CV cs.IR
keywords face-voice associationcross-modal embeddingconvex featuremulti-similarity losscross-modal attentionVoxCelebverificationretrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Face and voice of the same person look and sound different enough that ordinary embedding methods leave a large gap between the two modalities, producing many false positives and negatives. This paper claims the gap can be closed by forcing each identity’s face and voice features to live inside a common convex hull. It does so by constructing an intermediate “convex” feature that is a random convex combination of the attention-weighted face and voice embeddings, then training with multi-similarity loss so that same-identity points cluster tightly while different identities stay apart. Cross-modal attention further suppresses noisy dimensions that would otherwise blur class boundaries. On the large VoxCeleb benchmark the resulting embeddings improve verification, 1:2 matching and retrieval over both supervised and unsupervised prior art, reaching roughly 90 % AUC in the unconstrained verification setting. A sympathetic reader cares because the same geometric trick is simple, works without identity labels, and directly attacks the heterogeneity that has limited every earlier face-voice system.

Core claim

Same-identity face and voice embeddings can be made to occupy a shared convex region by introducing learnable intermediate convex features (random convex combinations of attention-weighted face and voice vectors) and optimizing them with multi-similarity metric learning; the resulting geometry simultaneously shrinks the cross-modal gap and enlarges inter-class separation, yielding state-of-the-art verification, matching and retrieval on VoxCeleb.

What carries the argument

Convex feature embedding (Eq. 3): an identity-specific intermediate vector formed as a normalized convex combination of attention-weighted face and voice embeddings, serving as an anchor that pulls the two modalities into the same convex hull.

Load-bearing premise

The method assumes that a person’s face and voice features can be usefully approximated by points inside one shared convex hull; if many people simply lack that cross-modal geometric resemblance, the intermediate anchors stop helping.

What would settle it

Measure intra-class cross-modal cosine distance on a held-out set of speakers known to have atypical face-voice pairings; if the distance does not drop below the best prior method and verification AUC does not rise, the convex-hull claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Unsupervised face-voice verification can reach nearly 90 % AUC without identity labels.
  • False-positive and false-negative rates in 1:2 matching and ranked retrieval fall once cross-modal gaps shrink.
  • The same convex-combination construction can be dropped into other audio-visual embedding pipelines that currently treat modalities as independent.
  • Gender-constrained and demographic-split evaluations remain ordered the same way, showing the gain is not an artifact of easy gender cues.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The residual failure clusters visible in t-SNE suggest a natural next test: condition the convex combination on paralinguistic or quality metadata so that atypical speakers are not forced into the same hull.
  • Because the construction is modality-agnostic, the identical intermediate-anchor idea could be tried on other heterogeneous pairs (e.g., sketch-photo or text-image) where convex geometry has not yet been imposed.
  • If the convexity assumption holds only for a statistical majority of people, practical systems may need an explicit “outlier” reject option rather than always returning a match.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a cross-modal embedding method for face–voice association that introduces identity-specific intermediate “convex” features formed as normalized combinations of attention-weighted face and voice embeddings (Eq. 3), together with a shared cross-modal attention MLP and multi-similarity metric learning (with iterative k-means in the unsupervised setting). The stated goal is to shrink modality heterogeneity so that same-identity face and voice features lie closer together while different identities remain separated. The method is evaluated on VoxCeleb (and AVSpeech) for verification, 1:2 matching, and retrieval under multiple splits, gender/nationality/age constraints, backbone variants, and ablations; reported unconstrained verification AUC is approximately 89.7%, with consistent gains over recent supervised and unsupervised baselines under a unified feature-extraction protocol.

Significance. If the empirical gains hold under the controlled backbone and split settings the authors use, the work is a useful incremental advance for face–voice association: ablations (Table 2), unified-backbone SOTA comparisons (Table 3), alternate splits (Table 5), alternate extractors (Table 4), AVSpeech (Table 6), intra-/inter-class distance analysis (Table 1, Fig. 5), and retrieval visualizations (Fig. 6) form a thorough experimental package. The training-time intermediate-anchor idea is simple and transferable. Credit is due for the breadth of controls and for openly discussing failure modes in §4.11. The main novelty claim is framed geometrically (convex hull / convex combinations); that framing is only partly accurate and should be tightened, but the empirical contribution remains of interest to the multi-modal biometrics community.

major comments (3)
  1. [§3.2.4, Eq. (3)] §3.2.4 and Eq. (3): the manuscript repeatedly asserts that same-identity face and voice features are embedded “in a convex hull” and that ¯yc is a convex combination lying inside that hull (Abstract; Fig. 2; contributions; §3.2.4). Because k(·) is applied per dimension and the denominator is an element-wise interpolation of attention scores, ¯yc is a coordinate-wise normalized weighted average, not a classical convex combination of the two vectors in R^{ne}. The resulting point is therefore not guaranteed to lie in conv{¯yf, ¯yv}. This is load-bearing for the paper’s geometric narrative and for the interpretation of Table 1 / Fig. 5 as “convexity.” Either prove a precise hull property under the stated operators, or replace the hull language with an accurate description (e.g., attention-normalized interpolation / training-time intermediate anchor) everywhere it appears.
  2. [§4.11, Fig. 4] §4.11 and Fig. 4: the method’s premise that same-identity face/voice pairs are well served by intermediate interpolants is acknowledged to fail for many identities (residual segregated clusters even under the full model). The SOTA attribution and the heterogeneity-reduction story would be stronger if the authors quantified how much of the AUC/mAP gain is concentrated on “easy” identities versus the hard residual cases they highlight. A short stratified analysis (e.g., gain vs. baseline face–voice distance bins, or vs. the cyan failure cluster type) would show whether convex embedding actually solves the hard heterogeneity cases claimed in the introduction, or mainly improves already-alignable pairs.
  3. [§3.2, §4.1] Inference protocol is under-specified relative to training. Convex features ¯yc and the clustering loop are training-time devices; test-time verification/matching/retrieval appear to use only (attention-weighted) face and voice embeddings. State explicitly what is used at test time, whether k(·) is applied at test time, and whether α is ever needed at inference. Without this, reproducibility of Tables 3–6 is incomplete and the role of “convex embedding” as a representation versus a training regularizer remains ambiguous.
minor comments (6)
  1. [Eq. (3)] Eq. (3) typesetting is ambiguous (scope of the fraction vs. ⊕). Rewrite with clear parentheses or a fraction bar and state that operations are element-wise.
  2. [§3.2.1, §4.1.2] Voice feature dimension is stated as 192 in §3.2.1 and 128 in §4.1.2; reconcile.
  3. [§4.9] Table 6 text says the proposed method shows “the lowest AUC and the highest EER,” which contradicts the table (highest AUC, lowest EER). Fix the sentence.
  4. [Fig. 3, §3.2.5] Fig. 3 overview is helpful but the path from ¯yc into k-means / multi-similarity loss is only described in prose; a clearer training-loop diagram or pseudocode would help.
  5. [§1, end] Mild self-extension of the authors’ SIGIR short paper [26] is disclosed; a short explicit “differences from [26]” paragraph (extra ablations, attention, multi-split/AVSpeech, geometric analysis) would help readers.
  6. Several typos and grammar issues (e.g., “network s”, “embedding s”, “cross-model” for cross-modal in places). A careful copy-edit pass is needed.

Circularity Check

0 steps flagged

No significant circularity: empirical cross-modal embedding method evaluated on held-out identities; mild disclosed self-extension of prior short paper does not force results by construction.

full rationale

The paper proposes a training objective (attention-weighted convex combinations of face/voice embeddings + multi-similarity loss + iterative k-means) and reports AUC/EER/ACC/mAP gains versus external baselines on public VoxCeleb and AVSpeech splits with unseen identities. Nothing in the derivation chain equates a claimed prediction or first-principles result to its own fitted inputs: Eq. 3 defines an intermediate training feature, not a quantity later reported as a discovery; Table 1 distances and all task metrics are measured post-training on held-out data. The sole self-citation ([26], the authors’ prior SIGIR short paper) is explicitly framed as an expansion that is re-evaluated with additional ablations and comparisons; it is not load-bearing for uniqueness or for the numerical claims. Geometric language around “convex hull” is an interpretive claim about the learned space, not a definitional identity that makes the SOTA numbers true by construction. Standard empirical ML paper; circularity burden is negligible.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The claim rests on standard deep metric-learning practice, pretrained unimodal extractors treated as fixed oracles, the modeling choice that same-identity face/voice pairs are positively associated and well served by linear interpolants in embedding space, and several hand-chosen training hyperparameters (embedding size, k-means k, α sampling, loss form). No physical entities are postulated; the ‘convex feature’ is a constructed training signal.

free parameters (4)
  • embedding dims nm, ne = 256
    Set to 256 by ablation; performance depends on this choice (Table 2).
  • k-means cluster count k = 1000
    Fixed at 1000 for unsupervised pseudo-labels; directly shapes the metric-learning targets.
  • convex mix weight α = U[0,1] (default)
    Sampled Uniform[0,1] per instance (or fixed 0.5 in ablation); controls the intermediate anchor location.
  • attention MLP width / dropout / LR schedule = LR=1e-3, batch=384, p=0.2
    Architecture and Adam 1e-3, decay 0.9, dropout 0.2, batch 384, 50 epochs chosen by authors; not derived.
axioms (5)
  • domain assumption Same-video or same-identity face–voice pairs are valid positives for association learning; different videos/identities are negatives (with reassignment after clustering).
    Stated in §3.1; standard in VoxCeleb face–voice work but false when videos share identity or when face/voice cues diverge.
  • domain assumption Frozen pretrained face (Inception-V1/VGGFace2) and voice (ECAPA-TDNN) features are sufficiently rich that only shallow embedding heads need training.
    §3.2.1; end-to-end fine-tuning said to overfit. Performance ceiling partly inherited from these extractors (Table 4).
  • ad hoc to paper A normalized convex combination of attention-weighted embeddings is a meaningful intra-class anchor that reduces modality gap.
    Core modeling choice Eq. (3) §3.2.4; motivated geometrically but not derived from a proven manifold theorem in the paper.
  • domain assumption Cosine/Euclidean geometry in the shared 256-D space plus multi-similarity loss adequately captures identity for verification, matching, and retrieval metrics.
    Standard metric-learning assumption used throughout §3–4.
  • standard math Linear algebra and backprop through elementwise ops and MLPs behave as usual.
    Implicit throughout architecture and training.
invented entities (2)
  • Convex feature embedding ȳ_c no independent evidence
    purpose: Serve as intermediate training anchor between face and voice so same-identity cross-modal features are pulled into a tighter region.
    Defined in Eq. (3); central novel construct. It is a computed training signal, not an external physical object; independent evidence is only the downstream metric gains in this paper.
  • Shared cross-modal attention network k(·) no independent evidence
    purpose: Per-dimension reweighting of face and voice embeddings before convex mixing to suppress non-discriminative dimensions.
    Lightweight shared MLP §3.2.3; attention is common, but the shared cross-modal use with convex mix is part of the proposed stack.

pith-pipeline@v1.2.0-daily-grok45 · 27681 in / 3533 out tokens · 70376 ms · 2026-07-31T16:56:01.497185+00:00 · methodology

0 comments
read the original abstract

Face-and-voice association learning is one of the most challenging tasks in deep learning. In this paper, we propose a simple but powerful cross-modal feature embedding method for the association of faces and voices. Previous work has studied cross-modal association tasks to establish the correlation between voice clips and facial images. These works have addressed cross-modal discrimination but underestimate the importance of handling heterogeneity in inter-modal features between audio and video, resulting in a lot of false positives and false negatives. To tackle the problem, the proposed method learns the embeddings of cross-modal features by making another feature exist between cross-modal features, facilitating the voice and face features of the same person to be embedded in a convex hull. Moreover, the incorporation of cross-modal attention mechanisms with convex embedding techniques represents a highly effective strategy for the attenuation of false positives and false negatives, accomplished via the minimization of inter-class discrepancies. We exhaustively evaluated our method for cross-modal verification, matching, and retrieval tasks on the large-scale VoxCeleb dataset. Extensive experimental results demonstrate that the proposed method achieves notable improvements over existing state-of-the-art methods.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Visual Cognition25(4- 6), 611–628 (2017)

    B¨ ulthoff, I., Newell, F.N.: Crossmodal priming of unfamiliar faces supports early interactions between voices and faces in person perception. Visual Cognition25(4- 6), 611–628 (2017)

    2017

  2. [2]

    Trends in Cognitive Sciences 17(6), 263–271 (2013)

    Yovel, G., Belin, P.: A unified coding strategy for processing faces and voices. Trends in Cognitive Sciences 17(6), 263–271 (2013)

    2013

  3. [3]

    Neuroimage 54(2), 1654–1661 (2011)

    Joassin, F., Maurage, P., Campanella, S.: The neural network sustaining the crossmodal processing of human gender from faces and voices: An fMRI study. Neuroimage 54(2), 1654–1661 (2011)

    2011

  4. [4]

    Computer Speech & Language 60, 101027 (2020)

    Nagrani, A., Chung, J.S., Xie, W., Zisserman, A.: VoxCeleb: Large-scale speaker verification in the wild. Computer Speech & Language 60, 101027 (2020)

    2020

  5. [5]

    In: Proceedings of the International Conference on Multimedia Retrieval, pp

    Chen, G., Zhang, D., Liu, T., Du, X.: Self-lifting: A novel framework for unsu- pervised voice-face association learning. In: Proceedings of the International Conference on Multimedia Retrieval, pp. 527–535. Association for Computing Machinery, New York, NY, USA (2022)

    2022

  6. [6]

    10452–10461

    Wu, C.-Y., Hsu, C.-C., Neumann, U.: Cross-modal perceptionist: Can face geom- etry be gleaned from voices? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10452–10461. IEEE, New York, NY, USA (2022)

    2022

  7. [7]

    In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp

    Saeed, M.S., Khan, M.H., Nawaz, S., Yousaf, M.H., Del Bue, A.: Fusion and orthogonal projection for improved face-voice association. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7057–7061. IEEE, New York, NY, USA (2022)

    2022

  8. [8]

    In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, pp

    Wang, R., Liu, X., Cheung, Y.-m., Cheng, K., Wang, N., Fan, W.: Learn- ing discriminative joint embeddings for efficient face and voice association. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1881–1884. Association for Computing Machinery, New York, NY, USA (2020)

    2020

  9. [9]

    In: Proceedings of the ACM International Conference 26 on Multimedia, pp

    Cheng, K., Liu, X., Cheung, Y.-m., Wang, R., Xu, X., Zhong, B.: Hearing like seeing: Improving voice-face interactions and associations via adversarial deep semantic matching network. In: Proceedings of the ACM International Conference 26 on Multimedia, pp. 448–455. Association for Computing Machinery, New York, NY, USA (2020)

    2020

  10. [10]

    In: Proceedings of the Asian Conference on Computer Vision, pp

    Kim, C., Shin, H.V., Oh, T.-H., Kaspar, A., Elgharib, M., Matusik, W.: On learning associations of faces and voices. In: Proceedings of the Asian Conference on Computer Vision, pp. 276–292. Springer, Berlin, Germany (2019)

    2019

  11. [11]

    In: Proceedings of the Digital Image Computing: Techniques and Applications, pp

    Nawaz, S., Janjua, M.K., Gallo, I., Mahmood, A., Calefati, A.: Deep latent space learning for cross-modal mapping of audio and visual signals. In: Proceedings of the Digital Image Computing: Techniques and Applications, pp. 1–7. IEEE, New York, NY, USA (2019)

    2019

  12. [12]

    Voice-Face Cross-modal Matching and Retrieval: A Benchmark

    Xiong, C., Zhang, D., Liu, T., Du, X.: Voice-face cross-modal matching and retrieval: A benchmark. arXiv (2019). https://doi.org/10.48550/arXiv.1911.09338 . http://arxiv.org/abs/1911.09338

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1911.09338 2019

  13. [13]

    In: Proceedings of the European Conference on Computer Vision, pp

    Nagrani, A., Albanie, S., Zisserman, A.: Learnable PINs: Cross-modal embeddings for person identity. In: Proceedings of the European Conference on Computer Vision, pp. 71–88. Springer, Berlin, Germany (2018)

    2018

  14. [14]

    In: Proceedings of the ACM International Conference on Multime- dia, pp

    Horiguchi, S., Kanda, N., Nagamatsu, K.: Face-voice matching using cross-modal embeddings. In: Proceedings of the ACM International Conference on Multime- dia, pp. 1011–1019. Association for Computing Machinery, New York, NY, USA (2018)

    2018

  15. [15]

    Disjoint Mapping Network for Cross-modal Matching of Voices and Faces

    Wen, Y., Ismail, M.A., Liu, W., Raj, B., Singh, R.: Disjoint mapping network for cross-modal matching of voices and faces. arXiv (2018). https://doi.org/10. 48550/arXiv.1807.04836 . https://arxiv.org/abs/1807.04836

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1807.04836 2018

  16. [16]

    In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R

    Sohn, K.: Improved deep metric learning with multi-class N-pair loss objective. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29. Curran Associates, Inc., New York, NY, USA (2016)

    2016

  17. [17]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823. IEEE, New York, NY, USA (2015)

    2015

  18. [18]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol

    Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1735–1742. IEEE, New York, NY, USA (2006)

    2006

  19. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wang, X., Han, X., Huang, W., Dong, D., Scott, M.R.: Multi-similarity loss with general pair weighting for deep metric learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5022–5030. IEEE, New York, NY, USA (2019) 27

    2019

  20. [20]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp

    Morgado, P., Vasconcelos, N., Misra, I.: Audio-visual instance discrimination with cross-modal agreement. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 12475–12486. IEEE, New York, NY, USA (2021)

    2021

  21. [21]

    In: Proceedings of the International Joint Conference on Artificial Intelligence, pp

    Zhu, B., Xu, K., Wang, C., Qin, Z., Sun, T., Wang, H., Peng, Y.: Unsuper- vised voice-face representation learning by cross-modal prototype contrast. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3787–3794. IJCAI Organization, California, USA (2022)

    2022

  22. [22]

    In: In Proceeding of the IEEE International Conference on Image Processing, New York, NY, USA, pp

    Chen, G., Zhang, D., Liu, T., Du, X.: Local-global contrast for learning voice- face representations. In: In Proceeding of the IEEE International Conference on Image Processing, New York, NY, USA, pp. 51–55 (2023). IEEE

    2023

  23. [23]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wei, X., Wu, J., Chen, Y., Wang, Z., Li, B.: Multi-modality cross attention network for image and sentence matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10941–10950 (2020)

    2020

  24. [24]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wei, X., Wu, J., Chen, Y., Wang, Z., Li, B.: Universal weighting metric learn- ing for cross-modal matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1415–1424 (2020)

    2020

  25. [25]

    arXiv preprint arXiv:2107.14724 (2021)

    Peng, D., Lei, Y., Li, W., Zhang, P., Guo, Y.: Sparse-to-dense feature match- ing: Intra and inter domain cross-modal learning in domain adaptation for 3D semantic segmentation. arXiv preprint arXiv:2107.14724 (2021)

    Pith/arXiv arXiv 2021

  26. [26]

    In: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp

    Kang, J., Kim, T., Park, Y.-h.: Convex feature embedding for face and voice asso- ciation. In: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2342–2346 (2024)

    2024

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wen, P., Xu, Q., Jiang, Y., Yang, Z., He, Y., Huang, Q.: Seeking the shape of sound: An adaptive framework for learning voice-face association. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16347–16356. IEEE, New York, NY, USA (2021)

    2021

  28. [28]

    Multimedia Systems 29(4), 2073–2083 (2023)

    Xu, X., Lv, G., Sun, Y., Hu, Y., Nian, F.: Hierarchical cross-modal contextual attention network for visual grounding. Multimedia Systems 29(4), 2073–2083 (2023)

    2073

  29. [29]

    International Journal of Multimedia Information Retrieval 13(1), 5 (2024)

    Wei, Y., Zheng, L., Qiu, G., Cai, G.: Cross-modal retrieval based on shared proxies. International Journal of Multimedia Information Retrieval 13(1), 5 (2024)

    2024

  30. [30]

    Multimedia Systems 30(2), 113 (2024) 28

    Liu, P., Liu, X.: Complementary expert balanced learning for long-tail cross- modal retrieval. Multimedia Systems 30(2), 113 (2024) 28

    2024

  31. [31]

    Multimedia Systems 30(5), 1–14 (2024)

    Li, Y., Tang, X., Lu, J., Huang, Y.: Dual graph-structured semantics multi- subspace learning for cross-modal retrieval. Multimedia Systems 30(5), 1–14 (2024)

    2024

  32. [32]

    Journal of Intelligent Information Systems, 1–25 (2024)

    Wang, Y., Xia, H., Liu, Y.: Cmc-mmr: multi-modal recommendation model with cross-modal correction. Journal of Intelligent Information Systems, 1–25 (2024)

    2024

  33. [33]

    In: Proceedings of the European Conference on Computer Vision, pp

    Lampert, C., Kroemer, O.: Weakly-paired maximum covariance analysis for multimodal dimensionality reduction and transfer learning. In: Proceedings of the European Conference on Computer Vision, pp. 566–579. Springer, Berlin, Germany (2010)

    2010

  34. [34]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol

    Kidron, E., Schechner, Y.Y., Elad, M.: Pixels that sound. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 88–95. IEEE, New York, NY, USA (2005)

    2005

  35. [35]

    In: Proceedings of the ACM International Conference on Multimedia, pp

    Li, D., Dimitrova, N., Li, M., Sethi, I.K.: Multimedia content processing through cross-modal association. In: Proceedings of the ACM International Conference on Multimedia, pp. 604–611. Association for Computing Machinery, New York, NY, USA (2003)

    2003

  36. [36]

    Journal of Machine Learning Research3, 1107–1135 (2003)

    Barnard, K., Duygulu, P., Forsyth, D., Freitas, N., Blei, D.M., Jordan, M.I.: Matching words and pictures. Journal of Machine Learning Research3, 1107–1135 (2003)

    2003

  37. [37]

    In: Pro- ceedings of the European Conference on Computer Vision, pp

    Duygulu, P., Barnard, K., Freitas, J.F., Forsyth, D.A.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In: Pro- ceedings of the European Conference on Computer Vision, pp. 97–112. Springer, Berlin, Germany (2002)

    2002

  38. [38]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Gordo, A., Larlus, D.: Beyond instance-level image retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6589–6598. IEEE, New York, NY, USA (2017)

    2017

  39. [39]

    arXiv (2014)

    Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embed- dings with multimodal neural language models. arXiv (2014). https://doi.org/10. 48550/arXiv.1411.2539 . http://arxiv.org/abs/1411.2539

    Pith/arXiv arXiv doi:10.48550/arxiv.1411.2539 2014

  40. [40]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Wang, L., Li, Y., Lazebnik, S.: Learning deep structure-preserving image-text embeddings. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5005–5013. IEEE, New York, NY, USA (2016)

    2016

  41. [41]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, pp

    Chen, Z.-D., Yu, W.-J., Li, C.-X., Nie, L., Xu, X.-S.: Dual deep neural net- works cross-modal hashing. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 274–281. AAAI Press, Washington, DC, USA (2018) 29

    2018

  42. [42]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Su, S., Zhong, Z., Zhang, C.: Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3027–3035. IEEE, New York, NY, USA (2019)

    2019

  43. [43]

    World Wide Web 22, 657–672 (2019)

    Xu, X., He, L., Lu, H., Gao, L., Ji, Y.: Deep adversarial metric learning for cross-modal retrieval. World Wide Web 22, 657–672 (2019)

    2019

  44. [44]

    IEEE Transactions on Multimedia 22(1), 174–187 (2019)

    Zhang, J., Peng, Y.: Multi-pathway generative adversarial hashing for unsuper- vised cross-modal retrieval. IEEE Transactions on Multimedia 22(1), 174–187 (2019)

    2019

  45. [45]

    IEEE Transactions on Multimedia 21(5), 1276–1288 (2018)

    Yu, E., Sun, J., Li, J., Chang, X., Han, X.-H., Hauptmann, A.G.: Adaptive semi-supervised feature selection for cross-modal retrieval. IEEE Transactions on Multimedia 21(5), 1276–1288 (2018)

    2018

  46. [46]

    IEEE Transactions on Multimedia 21(5), 1261–1275 (2018)

    Song, G., Wang, D., Tan, X.: Deep memory network for cross-modal retrieval. IEEE Transactions on Multimedia 21(5), 1261–1275 (2018)

    2018

  47. [47]

    IEEE Transactions on Cybernetics 50(6), 2400–2413 (2019)

    Xu, X., Lu, H., Song, J., Yang, Y., Shen, H.T., Li, X.: Ternary adversarial net- works with self-supervision for zero-shot cross-modal retrieval. IEEE Transactions on Cybernetics 50(6), 2400–2413 (2019)

    2019

  48. [48]

    IEEE Transactions on Circuits and Systems for Video Technology 30(4), 1173–1187 (2019)

    Chi, J., Peng, Y.: Zero-shot cross-media embedding learning with dual adversar- ial distribution network. IEEE Transactions on Circuits and Systems for Video Technology 30(4), 1173–1187 (2019)

    2019

  49. [49]

    In: Proceedings of the International Conference on Multimedia Retrieval, pp

    Albanie, S., Nagrani, A., Vedaldi, A., Zisserman, A.: Emotion recognition in speech using cross-modal transfer in the wild. In: Proceedings of the International Conference on Multimedia Retrieval, pp. 292–301. Association for Computing Machinery, New York, NY, USA (2018)

    2018

  50. [50]

    In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R

    Aytar, Y., Vondrick, C., Torralba, A.: SoundNet: Learning sound representations from unlabeled video. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29. Curran Associates, Inc., New York, NY, USA (2016)

    2016

  51. [51]

    In: Proceedings of the International Conference on Machine Learning, pp

    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the International Conference on Machine Learning, pp. 689–696. Omnipress, Madison, WI, USA (2011)

    2011

  52. [52]

    In: Pereira, F., Burges, C.J., Bottou, L., Weinberger, K.Q

    Srivastava, N., Salakhutdinov, R.R.: Multimodal learning with deep Boltzmann machines. In: Pereira, F., Burges, C.J., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25. Curran Associates, Inc., New York, NY, USA (2012)

    2012

  53. [53]

    In: Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), pp

    Zhou, Y., Wang, J., Li, X., Zheng, Z., Wu, Q., Loy, C.C.: Video entailment via 30 reaching a structure-aware cross-modal consensus. In: Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), pp. 6786–6795 (2023)

    2023

  54. [54]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2025)

    Zhang, Y., Li, Z., Wang, X., Wang, L., Yang, Y.: Learning probabilistic presence- absence evidence for weakly-supervised audio-visual event perception. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2025)

    2025

  55. [55]

    International Journal of Automation and Computing 18, 351–376 (2021)

    Zhu, H., Luo, M.-D., Wang, R., Zheng, A.-H., He, R.: Deep audio-visual learn- ing: A survey. International Journal of Automation and Computing 18, 351–376 (2021)

    2021

  56. [56]

    ACM Transactions on Graphics 33(4), 1–10 (2014)

    Davis, A., Rubinstein, M., Wadhwa, N., Mysore, G.J., Durand, F., Freeman, W.T.: The visual microphone: Passive recovery of sound from video. ACM Transactions on Graphics 33(4), 1–10 (2014)

    2014

  57. [57]

    In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp

    Wan, C.-H., Chuang, S.-P., Lee, H.-Y.: Towards audio to scene image synthesis using generative adversarial network. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 496–500. IEEE, New York, NY, USA (2019)

    2019

  58. [58]

    In: Proceedings of the on Thematic Workshops of ACM Multimedia, pp

    Chen, L., Srivastava, S., Duan, Z., Xu, C.: Deep cross-modal audio-visual gener- ation. In: Proceedings of the on Thematic Workshops of ACM Multimedia, pp. 349–357. Association for Computing Machinery, New York, NY, USA (2017)

    2017

  59. [59]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, pp

    Hao, W., Zhang, Z., Guan, H.: CMCGAN: A uniform framework for cross-modal visual-audio mutual generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6886–6893. AAAI Press, Washington, DC, USA (2018)

    2018

  60. [60]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Zhou, Y., Wang, Z., Fang, C., Bui, T., Berg, T.L.: Visual to sound: Generating natural sound for videos in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3550–3558. IEEE, New York, NY, USA (2018)

    2018

  61. [61]

    In: Proceedings of the INTERSPEECH 2015, pp

    Le Cornu, T., Milner, B.: Reconstructing intelligible audio speech from visual speech features. In: Proceedings of the INTERSPEECH 2015, pp. 3355–3359. International Speech Communication Association, Baixas, France (2015)

    2015

  62. [62]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing 25(9), 1751–1761 (2017)

    Le Cornu, T., Milner, B.: Generating intelligible audio speech from visual speech. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25(9), 1751–1761 (2017)

    2017

  63. [63]

    In: Solla, S., Leen, T., M¨ uller, K

    Hershey, J., Movellan, J.: Audio vision: Using audio-visual synchrony to locate sounds. In: Solla, S., Leen, T., M¨ uller, K. (eds.) Advances in Neural Information Processing Systems, vol. 12. MIT Press, Cambridge, MA, USA (1999)

    1999

  64. [64]

    In: Proceedings of the European Conference on Computer Vision, 31 pp

    Gao, R., Feris, R., Grauman, K.: Learning to separate object sounds by watching unlabeled video. In: Proceedings of the European Conference on Computer Vision, 31 pp. 35–53. Springer, Berlin, Germany (2018)

    2018

  65. [65]

    In: Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp

    Senocak, A., Oh, T.-H., Kim, J., Yang, M.-H., Kweon, I.S.: Learning to localize sound source in visual scenes. In: Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 4358–4366. IEEE, New York, NY, USA (2018)

    2018

  66. [66]

    In: Proceedings of the European Conference on Computer Vision, pp

    Tian, Y., Shi, J., Li, B., Duan, Z., Xu, C.: Audio-visual event localization in unconstrained videos. In: Proceedings of the European Conference on Computer Vision, pp. 247–263. Springer, Berlin, Germany (2018)

    2018

  67. [67]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp

    Nawaz, S., Saeed, M.S., Morerio, P., Mahmood, A., Gallo, I., Yousaf, M.H., Del Bue, A.: Cross-modal speaker verification and recognition: A multilingual perspective. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 1682–1691. IEEE, New York, NY, USA (2021)

    2021

  68. [68]

    In: Proceedings of the INTERSPEECH 2020, pp

    Tao, R., Das, R.K., Li, H.: Audio-visual speaker recognition with a cross-modal discriminative network. In: Proceedings of the INTERSPEECH 2020, pp. 2242–

    2020

  69. [69]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Nagrani, A., Albanie, S., Zisserman, A.: Seeing voices and hearing faces: Cross- modal biometric matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8427–8436. IEEE, New York, NY, USA (2018)

    2018

  70. [70]

    IEEE Transactions on Multimedia 24, 338–351 (2021)

    Zheng, A., Hu, M., Jiang, B., Huang, Y., Yan, Y., Luo, B.: Adversarial- metric learning for audio-visual cross-modal matching. IEEE Transactions on Multimedia 24, 338–351 (2021)

    2021

  71. [71]

    In: Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, pp

    Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: VGGFace2: A dataset for recognising faces across pose and age. In: Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, pp. 67–74. IEEE, New York, NY, USA (2018)

    2018

  72. [72]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9. IEEE, New York, NY, USA (2015)

    2015

  73. [73]

    In: Proceedings of the British Machine Vision Conference, pp

    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of the British Machine Vision Conference, pp. 41–14112. British Machine Vision Association, Durham, UK (2015). https://doi.org/10.5244/C.29.41

    doi:10.5244/c.29.41 2015

  74. [74]

    arXiv (2021)

    Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J., Chou, J.-C., Yeh, S.-L., Fu, S.-W., Liao, C.-F., Rastorgueva, E., Grondin, F., Aris, W., Na, H., Gao, Y., 32 De Mori, R., Bengio, Y.: SpeechBrain: A general-purpose speech toolkit. arXiv (2021). https://doi.org/10.48550/arXi...

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.04624 2021

  75. [75]

    In: Proceedings of the INTERSPEECH 2020, pp

    Desplanques, B., Thienpondt, J., Demuynck, K.: ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verifica- tion. In: Proceedings of the INTERSPEECH 2020, pp. 3830–3834. International Speech Communication Association, Baixas, France (2020)

    2020

  76. [76]

    In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp

    Xie, W., Nagrani, A., Chung, J.S., Zisserman, A.: Utterance-level aggregation for speaker recognition in the wild. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5791–5795. IEEE, New York, NY, USA (2019)

    2019

  77. [77]

    In: Proceedings of the INTERSPEECH 2018, pp

    Chung, J.S., Nagrani, A., Zisserman, A.: VoxCeleb2: Deep speaker recognition. In: Proceedings of the INTERSPEECH 2018, pp. 1086–1090. International Speech Communication Association, Baixas, France (2018)

    2018

  78. [78]

    arXiv (2020)

    Musgrave, K., Belongie, S., Lim, S.-N.: PyTorch metric learning. arXiv (2020). https://doi.org/10.48550/arXiv.2008.09164 . https://arxiv.org/abs/2008.09164

    Pith/arXiv arXiv doi:10.48550/arxiv.2008.09164 2020

  79. [79]

    In: Wallach, H., Larochelle, H., Beygelzimer, A., Alch´ e-Buc, F., Fox, E., Garnett, R

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H....

    2019

  80. [80]

    arXiv (2014)

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv (2014). https://doi.org/10.48550/arXiv.1412.6980 . https://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv doi:10.48550/arxiv.1412.6980 2014

Showing first 80 references.