Gramian Multimodal Representation Learning and Alignment

Danilo Comminiello; Eleonora Grassucci; Giordano Cicchetti; Luigi Sigillo

REVIEW 3 major objections 6 minor 4 cited by

Gramian Multimodal Representation Learning and Alignment

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that aligning any number of modalities reduces to shrinking the parallelotope their normalized embeddings span, and that training models to do so reaches state-of-the-art video-audio-text retrieval and audio-video…

desk verdict Neat Gramian-volume alignment idea with promising from-scratch results, but sign invariance and a pretraining confound keep the headline claims from fully landing. read the letter →

arxiv 2412.11959 v2 pith:YDRKKSY5 submitted 2024-12-16 cs.CV cs.AIcs.LG

Giordano Cicchetti , Eleonora Grassucci , Luigi Sigillo , Danilo Comminiello This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords Gramianmatrixparallelotopevolumemultimodalalignmentcontrastivelearningvideo-audio-textretrievalcosinesimilaritygeneralizationmodalitygaprepresentation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to replace pairwise, anchor-based multimodal alignment with a single joint measure. GRAM computes, for any 2 to $n$ normalized modality embeddings, the volume of the $k$-dimensional parallelotope they span — the square root of the determinant of their Gram matrix — and treats smaller volume as better semantic alignment. The authors build a contrastive loss on this volume, minimizing it for matched samples, and train a video-audio-text model that reports state-of-the-art Recall@1 on retrieval benchmarks such as MSR-VTT, DiDeMo, ActivityNet, and VATEX, plus gains on audio tasks, improving on its cosine-based counterpart by 4.9 to 7.4 points. The paper also shows that the same volume correlates with downstream performance ($\rho = 0.923$) and proposes GRAM as a metric for latent alignment. If correct, any existing multimodal model could swap cosine similarity for GRAM and align all modalities at once, with no architectural change.

What carries the argument

The central object is the Gramian volume: for $k$ unit-norm embedding vectors, the square root of the determinant of their $k \times k$ Gram matrix, $\mathrm{Vol}(v_1,\ldots,v_k) = \sqrt{\det G}$, which by a classical theorem is the volume of the $k$-dimensional parallelotope spanned by the vectors. The paper's machinery consists of using this single number both as the alignment measure and as the core of a contrastive loss, so one optimization objective couples all pairwise inner products at once. In the two-modality case the formula reduces to $\sin\theta$, a direct geometric generalization of cosine similarity, and for $k > 2$ it folds every pairwise term into one scalar, which is why the authors can claim simultaneous alignment of all modalities without a designated anchor.

What would settle it

Take a small two-modality retrieval benchmark, and for each query build a hard negative by flipping its true matching embedding to the antipode ($v \mapsto -v$), so cosine similarity is $-1$ while the Gramian volume $\sqrt{1 - \cos^2\theta} = 0$ is identical to the true positive's. If GRAM ranks these anti-aligned negatives as perfect matches, the claim that volume measures semantic alignment fails for that configuration; one could then look for this pattern arising in the trained model's own negatives.

Watch

Extended reading notes

Core claim

GRAM — the Gramian Representation Alignment Measure — is a similarity score defined across all modalities at once rather than pairwise. Given $k$ unit-norm modality embedding vectors $v_1,\ldots,v_k \in \mathbb{R}^n$, the Gram matrix $G$ has entries $\langle v_i, v_j \rangle$, and the volume of the $k$-dimensional parallelotope they span is $\mathrm{Vol} = \sqrt{\det G}$. The paper's central claim is that this volume is a faithful measure of joint semantic alignment: small volume means the modalities point in nearly the same direction in the shared embedding space, whereas cosine-based methods constrain only each modality against an anchor and leave non-anchor pairs unaligned. From this the authors derive a volume-based contrastive loss ($L_{D2A}$, $L_{A2D}$, plus a data-anchor matching loss $L_{DAM}$) that pulls matched modalities together and pushes negatives apart, and they prove the construction extends from $k=2$ up to $k=n$ modalities. Starting from the VAST backbone and pretraining briefly on a 150k-sample subset of VAST27M, the GRAM model reports state-of-the-art or improved Recall@1 on MSR-VTT, DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound, with gains of 4.9 to 7.4 points over the cosine-based baseline in retrieval, and the paper further shows that $1 - \mathrm{Vol}$ correlates with model performance ($\rho = 0.923$), proposing GRAM as a metric for latent alignment.

Load-bearing premise

The method rests on treating the volume spanned by the embeddings as a faithful measure of semantic alignment, but that equivalence is false for opposite-facing vectors: two unit vectors at angle $\pi$ have the same zero volume as two perfectly aligned ones, so the score cannot distinguish 'aligned' from 'anti-aligned'.

Editorial extensions

If this is right

Any downstream method that currently uses cosine similarity can swap in GRAM and align 2 to $n$ modalities jointly, without architectural modifications or added parameters.
In video-audio-text retrieval, the GRAM-trained model improves Recall@1 by 4.9 to 7.4 points over its cosine-based counterpart across zero-shot and fine-tuning settings on MSR-VTT, DiDeMo, ActivityNet, and VATEX.
Adding modalities one at a time — audio, then subtitles, then depth — raises zero-shot text-to-video Recall@1 on MSR-VTT from 52.8 to 55.3, supporting the claim that joint higher-dimensional alignment captures richer semantics than pairwise anchoring.
GRAM also functions as a performance metric: across LanguageBind, VAST, and the GRAM model on MSR-VTT, the rescaled $1-\mathrm{Vol}$ score correlates with downstream Recall@1 at $\rho = 0.923$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The volume measure is sign-invariant: for two unit vectors it equals $\sin\theta$, so perfectly aligned ($\theta=0$) and exactly anti-aligned ($\theta=\pi$) vectors get the same zero score, which means a negative pair lying in opposite directions would be scored as a perfect match by the GRAM loss in Eqs. (5)–(6) — a case the paper does not discuss.
The modality-gap measurements in the paper's Table 7 (larger inter-modality centroid distances after GRAM training) may be the flip side of that invariance: the loss can satisfy a small volume by pushing modality clusters to opposite sides of the hypersphere rather than by merging them, and neither the volume value nor the reported gap would reveal which happened.
A direct way to isolate the mechanism would be to train the same backbone with a cosine-based three-way objective that minimizes each modality's distance to the mean embedding of the others; if GRAM's gains persist, the advantage comes from joint alignment itself, not from the specific volume formulation.
Because the construction is defined for any $k \le n$, GRAM could in principle align non-sensory embedding sets — multiple languages, multiple views, or heterogeneous feature groups — as long as the sign-invariance caveat is handled, though the paper only demonstrates sensory modalities.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper has a genuinely new idea — replace pairwise cosine similarity with the determinant of the Gram matrix, i.e., the volume of the parallelotope spanned by n modality vectors, as a multi-way alignment score. The math is standard but cleanly applied, the loss is simple, and the paper ships code, pretrained models, and a from-scratch ablation that isolates the loss from the pretraining confound. Worth a serious referee. But there are two soft spots that need airing.

The first is the sign-invariance problem, and it is real. Their own Appendix A.3 shows that for two unit vectors, Vol = |sin θ|, because the Gram determinant is unchanged by flipping any one vector (det(DGD) = det(G)). The loss in Eqs. 5–6 uses exp(-Vol/τ) as the alignment score, so an antiparallel negative pair is scored identically to an aligned positive pair. The paper never mentions this, and it directly contradicts the claim that "lower volume means closer vectors." It also makes Section 3.5's use of GRAM as a model-performance metric questionable: a model with one modality globally sign-flipped would get a perfect GRAM score while being anti-aligned in the cosine sense the paper says it replaces. In practice, gradient descent from a sensible initialization may avoid the flipped basin, so the empirical retrieval numbers can still be valid. But the measure itself, as a measure of alignment, is weaker than advertised.

The second soft spot is the main comparison. The paper takes V AST pretrained models, adds one extra epoch of pretraining on 150k samples with the GRAM loss, and compares against the original, off-the-shelf V AST. That is not a controlled comparison; part of the 4.9–7.4 point gain could come from the extra training, not the loss. The from-scratch ablation in Table 6 is much fairer — it trains both the pairwise cosine loss and the GRAM loss from scratch with the same budget, and GRAM wins. That is the strongest evidence in the paper, and it does support the core claim. But the abstract's headline numbers come from the confounded setting.

The circularity concern is minor: the model is trained with the volume loss and then evaluated with a volume-based metric in Section 3.5, but the downstream R@1 numbers are independent, so this is not load-bearing.

Bottom line: the idea is promising and the from-scratch results suggest it has real value, but the sign invariance is a genuine geometric flaw that the authors need to address — either by modifying the measure (oriented volume, or adding a cosine term for the two-modality case) or by discussing why it does not matter empirically. The pretraining confound needs a matched baseline. I would send this to peer review, not desk reject, and I would ask the reviewers to push on both issues.

Referee Report

3 major / 6 minor

Summary. The paper introduces GRAM, a geometric alignment measure for multimodal embeddings. For k normalized modality vectors, GRAM is defined as the volume of the k-dimensional parallelotope they span, computed as the square root of the Gram determinant, and is proposed as a replacement for cosine similarity in multimodal contrastive learning and retrieval. The authors derive the k=2 reduction to |sin θ|, introduce a GRAM-based contrastive loss (Eqs. 5-6) combined with a data-anchor matching loss, pretrain a VAST-initialized model on a 150k subset, and report improved zero-shot and fine-tuned R@1 on MSR-VTT, DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound. They also propose GRAM as a model-performance metric, supported by a correlation plot in Fig. 3.

Significance. If the central claim were fully supported, GRAM would be a valuable drop-in replacement for pairwise cosine similarity, scaling naturally from 2 to n modalities with consistent gains; the paper provides code, model releases, a formal volume derivation, and from-scratch ablations in Appendix B.2, which are strengths. However, the proposed measure is invariant to flipping any modality vector, so it cannot distinguish aligned from anti-aligned embeddings; this directly contradicts the claim that low volume means modality vectors are 'closer' and weakens the use of GRAM as a semantic alignment measure. In addition, the GRAM-as-metric analysis is partly circular for the model trained with the same loss, and the headline comparisons against VAST are not controlled for additional pretraining. The from-scratch ablation in Table 6 is the soundest evidence for the method and should be central.

major comments (3)

[3.3 / A.3 (Eq. 18)] The volume Vol(v1,...,vk)=sqrt(det G) is invariant under flipping any single vector vi to -vi, because the Gram matrix transforms as G' = D G D with D a diagonal matrix of ±1 entries, so det G' = det G. For k=2, Eq. (18) gives Vol = |sin θ|, so θ=0 and θ=π produce identical zero volume. Consequently, anti-parallel embeddings are scored as perfectly aligned, and the statement in Section 3.3 that 'the lower the volume, the closer the modality vectors are' is not correct: maximally separated (opposite) unit vectors also minimize the volume. This invariance propagates into the contrastive loss in Eqs. (5)-(6): a negative pair whose vectors are anti-parallel receives the same exp(-Vol/τ) score as a positive aligned pair, and the GRAM terms in L_TOT provide no training signal to choose between a modality vector and its negation. The paper never states or justifies this sign invariance, yet it is central to the claim that GRAM provides 'more meaningful alignment' than cosine similarity. The authors need to break this symmetry, for example by combining volume with signed dot products, or to provide an explicit justification and empirical evidence that sign-consistent solutions are enforced by the optimization.
[3.5 / Fig. 3] The claim that GRAM is a reliable performance metric is weakened by a self-referential loop. The GRAM model is trained to minimize exactly the quantity that is then used as the metric, so a low GRAM value for this model is expected by construction. The correlation of ρ=0.923 is computed over three models, one of which (the GRAM model) is optimized for the metric; this does not establish that GRAM predicts performance on arbitrary multimodal models. I recommend recomputing the correlation on a set of models that were not trained with the GRAM objective, and reporting the correlation separately for models trained with and without GRAM. Without this, Section 3.5's claim that GRAM 'can serve as a metric for evaluating large multimodal models' is not supported.
[Tables 1-3 / Section 4.1] The main comparisons against the VAST baseline are not controlled. The GRAM model is initialized from VAST pretrained weights and then further pretrained for one epoch on a 150k-sample subset of VAST27M with the GRAM loss, while the VAST row receives no such additional pretraining. Therefore the reported gains of +4.9 to +7.4 R@1 in Tables 1-2 and the gains in Table 3 cannot be attributed solely to the GRAM loss. The from-scratch ablation in Table 6 (Appendix B.2), which compares the GRAM loss with cosine TV-TA losses under identical training, is the correct controlled experiment and should be moved to the main text or at least carefully discussed alongside the headline numbers.

minor comments (6)

[Section 4.4] For two vectors the volume is the area of a parallelogram, not a triangle; the sentence 'the volume computation degenerates to the area of the triangle' should be corrected.
[Section 3.2] In the k > n case, the sentence 'the volume of the k-parallelotope is still positive but equal to zero' is contradictory; it should read 'is zero'.
[Appendix A.1] In the k < n case, the proof text says 'where k ≥ n'; this should be 'where k < n'.
[Section 3.4] The superscript/subscript notation in Eqs. (5)-(6) is not defined consistently: the text says m_x^y refers to the embedding of the x-th modality of the j-th sample, but the equations use m_i^j and m_j^k without a clear statement of which index is the sample and which is the modality.
[Reproducibility statement] The reproducibility statement says pretrained models 'will be released after reviewing process,' while the abstract and introduction state they are available; please update for consistency.
[Throughout] Minor typos: 'k-dimesnional' in Section 3.4, 'donwstream' in Section 4.4, and 'GRAMIAN' in the Appendix A title.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the Gramian-volume loss is not fitted to downstream R@1 and the headline results are measured on external benchmarks; the only self-referential piece is a minor confound in the Section 3.5 metric-correlation plot.

full rationale

The paper's central claim is that minimizing the Gramian volume (Eq. 4) of modality embeddings, via the contrastive losses in Eqs. 5-6, produces better multimodal alignment, measured by R@1 on MSR-VTT, DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound. Those downstream metrics are external to the loss, so the 'prediction' does not reduce to the training objective by construction. The Gramian volume itself is a standard geometric quantity (determinant of the Gram matrix), not fitted to the target labels, and the loss definitions are explicit rather than imported from an unverified self-citation. No uniqueness theorem or prior-work ansatz is used to forbid alternatives. The nearest thing to circularity is Section 3.5 and Figure 3, where GRAM is proposed as a performance metric and correlated with R@1 across LanguageBind, VAST, and the GRAM-based model; the GRAM-based model was trained to minimize exactly this volume, so its high 1-GRAM value is partly by construction. That is a self-referential confound in the metric-validation evidence, but it is not a formal reduction because R@1 and 1-GRAM are not equal by any equation and the correlation is empirical. Separately, Appendix A.3's Vol = |sin(theta)| (Eq. 18) shows the volume cannot distinguish aligned from anti-aligned vectors; this is a real soundness limitation of volume as a semantic alignment measure, but it is a correctness issue, not a circularity. Finally, Appendix B.3 honestly states that no mathematical proof links the larger modality gap to downstream performance; this is a stated limitation, not a hidden circular step. Overall, the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the standard theorem that the Gram determinant equals squared volume, on the normalisation of embeddings to unit norm, on the assumption that k < n, and on the unstated and unproven premise that low volume corresponds to semantic alignment. The sign-flip invariance of the volume is an additional hidden assumption that can mis-score anti-aligned embeddings. No free parameters beyond standard training hyperparameters are fitted to make the derivation work.

free parameters (2)

temperature τ = learnable
Learnable scaling parameter in the GRAM contrastive loss (Eqs. 5-6). Standard in contrastive learning, but is a fitted model parameter.
loss weight λ = 0.1
Weight for the data-anchor matching loss in Eq. 8, taken from Chen et al. (2023c). Not optimized by the authors.

assumptions (5)

standard math Gram determinant equals squared parallelotope volume
Standard linear algebra theorem (Gantmacher 1959), used in Theorem 1 and throughout.
domain assumption Embedding vectors are normalized to unit norm
Section 3.4: 'we normalize them. In this way... norm... is always 1'. Needed for volume to indicate alignment.
domain assumption Number of modalities k is less than embedding dimension n
Section 3.2: 'in current neural embedding models, k << n'. If k > n, the Gramian is singular and uninformative.
domain assumption Semantic alignment of modalities is equivalent to small geometric volume spanned by their embeddings
Central premise; stated as intuition in Section 3.3 and Fig. 1. Untested empirically beyond correlation plot.
ad hoc to paper Anti-aligned embeddings are not problematic for the measure
Volume is invariant under sign flips (Appendix A.3: Vol = sin θ, so θ and π-θ give identical volume). The paper does not justify ignoring this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Gramian Multimodal Representation Learning and Alignment." pith.science (2026). https://pith.science/paper/YDRKKSY5

@misc{pith2026241211959,
  author       = {Pith},
  title        = {Pith review of: Gramian Multimodal Representation Learning and Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDRKKSY5}},
  note         = {Machine review of arXiv:2412.11959}
}

abstract

Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligning pairs of modalities via contrastive learning, their solutions are unsuitable when scaling to multiple modalities. These models typically align each modality to a designated anchor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional approach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above-mentioned limitations. GRAM learns and then aligns $n$ modalities directly in the higher-dimensional space in which modality embeddings lie by minimizing the Gramian volume of the $k$-dimensional parallelotope spanned by the modality vectors, ensuring the geometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to $n$ modalities and providing more meaningful alignment with respect to previous similarity measures. The novel GRAM-based contrastive loss function enhances the alignment of multimodal models in the higher-dimensional embedding space, leading to new state-of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification. The project page, the code, and the pretrained models are available at https://ispamm.github.io/GRAM/.

Figures

Figures reproduced from arXiv: 2412.11959 by the authors.

**Figure 2.** GRAM-based model architecture. Class tokens from each modality are involved in shap [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** The proposed GRAM similarity is strongly correlated (ρ = 0.923) with large multimodal models performance in downstream tasks. As GRAM is a measure of the multimodal embedding space alignment, it can also serve as a metric for evaluating large multimodal models. Indeed, the more aligned the multimodal latent space, the better the model will perform in downstream tasks, as it possesses rich semantic information. W… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: t-SNE visualization on VGGSound of VAST, cosine-based, (left) and GRAM (right) latent [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Multimodal V2T/T2V, V2T/T2V, Gramian Value and training loss of GRAM trained with [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

**Figure 6.** Figure 6: Confusion matrices of cosine-based approach and our proposed GRAM. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data
cs.CV 2026-07 accept novelty 7.0 of 10

A camera-trap TVR benchmark of 135 ethology queries plus an interpretable SALMA-to-JSON plus constrained-LLM-parser pipeline yields 34% set F1, beating zero-shot VLMs at 18%.
SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding
cs.SD 2025-06 conditional novelty 6.0 of 10

SLAP trains joint music-text embeddings with a BYOL-style, negative-free loss and outperforms CLAP on music-text retrieval, zero-shot classification, and several downstream MIR tasks while reducing the modality gap.
Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
cs.CV 2026-07 conditional novelty 5.0 of 10

A two-stage alignment framework that first fuses visual modalities (RGB, flow, skeleton) then introduces text, achieving 21% SRCC improvement on a new clinical AQA dataset and gains on two public benchmarks.
VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization
cs.CV 2025-05 reject novelty 4.0 of 10

VISTA adds a position-weighted, parameter-free L2 alignment loss between text token embeddings and visual hidden states to MLLM training, reporting modest benchmark gains but relying on an invalid information-theoretic proof.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages · cited by 4 Pith papers

[1]

Therefore, equation 17 becomes: V ol= p det(G) = p 1 − cos2(θ) = q sin2(θ) = sin(θ)

The Gram matrix G is given by: G = ⟨v⊤ 1 v1⟩ ⟨v⊤ 1 v2⟩ ⟨v⊤ 2 v1⟩ ⟨v⊤ 2 v2⟩ (14) Then, compute the determinant of the Gram matrix: det(G) =⟨v⊤ 1 v1⟩⟨v⊤ 2 v2⟩ − ⟨v⊤ 1 v2⟩2 (15) The volume of the k-dimensional parallelotope spanned by the modalities v1, v2 is: V ol= p det(G) = q ⟨v⊤ 1 v1⟩⟨v⊤ 2 v2⟩ − ⟨v⊤ 1 v2⟩2 (16) Given that v1 and v2 have norm equal to 1, ...

work page 2016
[3]

,vk) = p det G(v1,

The determinant of the Gram matrix, also called the Gramian, is the square of the volume of the k-dimensional parallelotope formed by the vectors Gantmacher (1959): V ol(v1, . . . ,vk) = p det G(v1, . . . ,vk). (9) Note the similarity to the norm of a vector; in fact, v(v) = ∥v∥, so the Gramian of a single vector is the length, i.e., the 1-dimensional vol...

work page 1959
[4]

Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all

Yuanhuiyi Lyu, Xueye Zheng, Xueye Kim, and Lin Wang. Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all. ArXiv preprint: arXiv:2405.16108,

arXiv
[5]

Learning audio-video modalities from image captions

12 Published as a conference paper at ICLR 2025 Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Man ´en, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. InEuropean Conference on Computer Vision,

work page 2025
[6]

Audio retrieval with natural language queries

Andreea-Maria Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries. arXiv preprint arXiv:2105.02192,

arXiv
[7]

Learning temporally consistent video depth from video diffusion priors

Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. ArXiv preprint: arXiv:2406.01493,

arXiv
[9]

Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric

Toshimitsu Uesaka, Taiji Suzuki, Yuhta Takida, Chieh-Hsin Lai, Naoki Murata, and Yuki Mitsufuji. Understanding multimodal contrastive learning through pointwise mutual information. ArXiv preprint: arXiv:2404.19228,

work page Pith review arXiv
[10]

(2022) T-V 33.3 66.7 34.0 68.7 31.9 72.0 - - OmniVL Wang et al

Zero-Shot T2V Retrieval MSR-VTT DiDeMo ActivityNet V ATEX Modality R@1 R@10 R@1 R@10 R@1 R@10 R@1 R@10 UMT Liu et al. (2022) T-V 33.3 66.7 34.0 68.7 31.9 72.0 - - OmniVL Wang et al. (2022a) T-V 42.0 73.0 40.6 74.3 - - - - UMT-L Li et al. (2023) T-V 40.7 71.8 48.6 79.0 41.9 - - - TVTSv2 Zeng et al. (2023) T-V 38.2 73.2 34.6 71.5 - - - - ViCLIP Wang et al. ...

work page 2022

Show all 19 references

[11]

InternVideo2: Scaling video foundation models for mul- timodal video understanding

Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. InternVideo2: Scaling video foundation models for mul- timoda...

arXiv
[12]

TVTSv2: Learning out-of-the-box spatiotemporal visual representations at scale

Ziyun Zeng, Yixiao Ge, Zhan Tong, Xihui Liu, Shutao Xia, and Ying Shan. TVTSv2: Learning out-of-the-box spatiotemporal visual representations at scale. ArXiv preprint: arXiv:2305.14173,

arXiv
[13]

Given v1,

14 Published as a conference paper at ICLR 2025 A T HEORETICAL APPENDIX A.1 T HE GRAMIAM COMPUTES THE VOLUME OF ANY k-DIMENSIONAL POLYTOPE Theorem 1 : Volume of the k-dimensional parallelotope. Given v1, . . . ,vk be k vectors in Rn forming a k-dimensional parallelotope. v1, ....

2025
[16]

Recall at 1 is shown for both Text-to-Video (T2V) and Video-to-Text (V2T) tasks training from scratch on MSR-VTT and ActivityNet datasets

17 Published as a conference paper at ICLR 2025 Table 6: Ablation study on loss functions. Recall at 1 is shown for both Text-to-Video (T2V) and Video-to-Text (V2T) tasks training from scratch on MSR-VTT and ActivityNet datasets. MSR-VTT ActivityNet TV TA D2A A2D DAM T2V V2T T...

2022
[17]

A dog is barking

Our GRAM model outperforms every method in the very large set of comparison methods in T2V and V2T both in zero-shot and fine-tuning scenarios. More interestingly, the proposed method always achieves very 19 Published as a conference paper at ICLR 2025 Video 1 = Video 2 = Vide...

2025
[19]

(2022) T-V 33.3 66.7 34.0 68.7 31.9 72.0 - - OmniVL Wang et al

Zero-Shot V2T Retrieval MSR-VTT DiDeMo ActivityNet V ATEX Modality R@1 R@10 R@1 R@10 R@1 R@10 R@1 R@10 UMT Liu et al. (2022) T-V 33.3 66.7 34.0 68.7 31.9 72.0 - - OmniVL Wang et al. (2022a) T-V 34.6 66.6 33.3 68.5 - - - - UMT-L Li et al. (2023) T-V 40.7 - 24.9 - 39.4 - - - TVT...

2022
[2019]

Internvideo: General video foundation models via generative and discriminative learning

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lear...

2025 arXiv
[2020]

BEATs: Audio pre-training with acoustic tokenizers

Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Interna- tional Conference on Machine Learning, pp. 5178–5193, 2023a. Sihan Chen, Xingjian He, Longteng G...

arXiv
[2021]

The kinetics human action video dataset

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950,

arXiv
[2023]

EV A-CLIP: Improved training techniques for clip at scale

Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: Improved training techniques for clip at scale. ArXiv preprint: arXiv:2303.15389,

arXiv
[2024]

CLAP: learning audio concepts from natural language supervision

11 Published as a conference paper at ICLR 2025 Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: learning audio concepts from natural language supervision. In IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. ...

2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Therefore, equation 17 becomes: V ol= p det(G) = p 1 − cos2(θ) = q sin2(θ) = sin(θ)

The Gram matrix G is given by: G = ⟨v⊤ 1 v1⟩ ⟨v⊤ 1 v2⟩ ⟨v⊤ 2 v1⟩ ⟨v⊤ 2 v2⟩ (14) Then, compute the determinant of the Gram matrix: det(G) =⟨v⊤ 1 v1⟩⟨v⊤ 2 v2⟩ − ⟨v⊤ 1 v2⟩2 (15) The volume of the k-dimensional parallelotope spanned by the modalities v1, v2 is: V ol= p det(G) = q ⟨v⊤ 1 v1⟩⟨v⊤ 2 v2⟩ − ⟨v⊤ 1 v2⟩2 (16) Given that v1 and v2 have norm equal to 1, ...

work page 2016

[2] [3]

,vk) = p det G(v1,

The determinant of the Gram matrix, also called the Gramian, is the square of the volume of the k-dimensional parallelotope formed by the vectors Gantmacher (1959): V ol(v1, . . . ,vk) = p det G(v1, . . . ,vk). (9) Note the similarity to the norm of a vector; in fact, v(v) = ∥v∥, so the Gramian of a single vector is the length, i.e., the 1-dimensional vol...

work page 1959

[3] [4]

Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all

Yuanhuiyi Lyu, Xueye Zheng, Xueye Kim, and Lin Wang. Omnibind: Teach to build unequal-scale modality interaction for omni-bind of all. ArXiv preprint: arXiv:2405.16108,

arXiv

[4] [5]

Learning audio-video modalities from image captions

12 Published as a conference paper at ICLR 2025 Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Man ´en, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. InEuropean Conference on Computer Vision,

work page 2025

[5] [6]

Audio retrieval with natural language queries

Andreea-Maria Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries. arXiv preprint arXiv:2105.02192,

arXiv

[6] [7]

Learning temporally consistent video depth from video diffusion priors

Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. ArXiv preprint: arXiv:2406.01493,

arXiv

[7] [9]

Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric

Toshimitsu Uesaka, Taiji Suzuki, Yuhta Takida, Chieh-Hsin Lai, Naoki Murata, and Yuki Mitsufuji. Understanding multimodal contrastive learning through pointwise mutual information. ArXiv preprint: arXiv:2404.19228,

work page Pith review arXiv

[8] [10]

(2022) T-V 33.3 66.7 34.0 68.7 31.9 72.0 - - OmniVL Wang et al

Zero-Shot T2V Retrieval MSR-VTT DiDeMo ActivityNet V ATEX Modality R@1 R@10 R@1 R@10 R@1 R@10 R@1 R@10 UMT Liu et al. (2022) T-V 33.3 66.7 34.0 68.7 31.9 72.0 - - OmniVL Wang et al. (2022a) T-V 42.0 73.0 40.6 74.3 - - - - UMT-L Li et al. (2023) T-V 40.7 71.8 48.6 79.0 41.9 - - - TVTSv2 Zeng et al. (2023) T-V 38.2 73.2 34.6 71.5 - - - - ViCLIP Wang et al. ...

work page 2022