{"id":"84734ee3-309a-4bca-b818-3559c2b034ea","arxiv_id":"2412.11959","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"GRAM replaces cosine similarity with the Gramian volume of the parallelotope formed by multiple modality embeddings, and a volume-based contrastive loss improves multimodal retrieval and classification.","lead":"A new loss function aligns any number of modality embeddings by minimizing the volume of the geometric shape they span. The authors report state-of-the-art results in video-audio-text retrieval and audio-video classification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sign invariance of the parallelotope volume makes GRAM unable to distinguish aligned from anti-aligned modalities; a model with one modality's embeddings globally sign-flipped would receive identical GRAM loss and retrieval scores.","rationale":"The reader's weakest assumption is exactly the sign-invariance of the volume-based alignment measure, and I agree that it is the most load-bearing concern. The paper's central novelty is an alignment measure that is supposed to generalize cosine similarity to n modalities while providing 'geometric alignment' and 'more meaningful alignment'. But for k=2, the measure is |sin theta|, which is zero for both theta=0 and theta=pi; for k>2, flipping any one vector leaves the Gram determinant unchanged. Thus the GRAM loss treats perfectly anti-aligned modalities as perfectly aligned, and the GRAM metric cannot detect a global sign flip of one modality. This is not an edge case outside the method's scope: it follows from the definition of the volume of a parallelotope, and the loss provides no gradient to select a consistent orientation across modalities. The empirical results, including the from-scratch ablation in Table 6, suggest the training procedure can still produce useful representations in practice, likely because random initialization and the data distribution break the symmetry. But the theoretical claim that GRAM 'ensures geometric alignment' is not supported, and the claim that GRAM provides a faithful quantitative metric of alignment is directly contradicted by the invariance. The reader's conditional verdict remains appropriate: the paper should address this sign-invariance, either by justifying that it is harmless in practice or by modifying the measure to break the symmetry, and it should also provide a matched-pretraining comparison. I see no need to change the verdict to ACCEPT or REJECT on the basis of this analysis alone.","tokens_in":22578,"tokens_out":5282,"duration_ms":54495,"concrete_test":"Run a controlled experiment on a toy two-modality dataset where every positive pair has dot product -1 and negatives are random. Train with the GRAM contrastive loss (Eqs. 5-6) from an identity-like initialization; if the loss is already at its minimum (volume 0 for all positives) with no parameter updates, the measure provably cannot distinguish anti-alignment from alignment. As a model-level check, take a released GRAM checkpoint, multiply the audio encoder's output projections by -1, and evaluate MSR-VTT retrieval with (a) GRAM volume and (b) cosine similarity. GRAM scores must be bit-identical by the invariance, while cosine R@1 should drop materially; this demonstrates that the learned space is not aligned in the cosine sense that the paper claims to replace.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central measure is Vol(v1,...,vk) = sqrt(det G), and for k=2 with unit vectors, Vol = |sin(theta)| (Appendix A.3, Eq. 18). This is invariant under flipping any single vector v -> -v: the Gram matrix transforms as G' = D G D with D diagonal having one -1, so det(G') = det(G). Therefore antiparallel vectors (theta = pi) have exactly the same volume as parallel vectors (theta = 0), both giving zero volume. The GRAM loss in Eqs. 5-6 uses exp(-Vol/tau) as the alignment score, so an antiparallel negative pair is scored as perfectly aligned, and a positive pair can be satisfied by making a modality's embeddings globally antiparallel to the others. Because the loss is invariant to per-modality sign flips, there is no training signal to choose the sign of, say, the audio encoder relative to text and video; a converged solution with audio embeddings globally flipped is equally optimal under L_TOT and yields identical GRAM-based retrieval, even though the modalities are anti-aligned in the cosine sense that GRAM claims to replace. This directly contradicts the paper's claim that 'the lower the volume, the closer the modality vectors are' and undermines the use of GRAM as a model-performance metric in Section 3.5, since a model with anti-aligned modalities would receive a perfect GRAM score. The pretraining confound in the main tables is also a concern, but the sign invariance is a property of the measure itself and is therefore the more load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GRAM, a geometric alignment measure for multimodal embeddings. For k normalized modality vectors, GRAM is defined as the volume of the k-dimensional parallelotope they span, computed as the square root of the Gram determinant, and is proposed as a replacement for cosine similarity in multimodal contrastive learning and retrieval. The authors derive the k=2 reduction to |sin θ|, introduce a GRAM-based contrastive loss (Eqs. 5-6) combined with a data-anchor matching loss, pretrain a VAST-initialized model on a 150k subset, and report improved zero-shot and fine-tuned R@1 on MSR-VTT, DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound. They also propose GRAM as a model-performance metric, supported by a correlation plot in Fig. 3.","tokens_in":22877,"tokens_out":10565,"duration_ms":99782,"significance":"If the central claim were fully supported, GRAM would be a valuable drop-in replacement for pairwise cosine similarity, scaling naturally from 2 to n modalities with consistent gains; the paper provides code, model releases, a formal volume derivation, and from-scratch ablations in Appendix B.2, which are strengths. However, the proposed measure is invariant to flipping any modality vector, so it cannot distinguish aligned from anti-aligned embeddings; this directly contradicts the claim that low volume means modality vectors are 'closer' and weakens the use of GRAM as a semantic alignment measure. In addition, the GRAM-as-metric analysis is partly circular for the model trained with the same loss, and the headline comparisons against VAST are not controlled for additional pretraining. The from-scratch ablation in Table 6 is the soundest evidence for the method and should be central.","major_comments":[{"comment":"The volume Vol(v1,...,vk)=sqrt(det G) is invariant under flipping any single vector vi to -vi, because the Gram matrix transforms as G' = D G D with D a diagonal matrix of ±1 entries, so det G' = det G. For k=2, Eq. (18) gives Vol = |sin θ|, so θ=0 and θ=π produce identical zero volume. Consequently, anti-parallel embeddings are scored as perfectly aligned, and the statement in Section 3.3 that 'the lower the volume, the closer the modality vectors are' is not correct: maximally separated (opposite) unit vectors also minimize the volume. This invariance propagates into the contrastive loss in Eqs. (5)-(6): a negative pair whose vectors are anti-parallel receives the same exp(-Vol/τ) score as a positive aligned pair, and the GRAM terms in L_TOT provide no training signal to choose between a modality vector and its negation. The paper never states or justifies this sign invariance, yet it is central to the claim that GRAM provides 'more meaningful alignment' than cosine similarity. The authors need to break this symmetry, for example by combining volume with signed dot products, or to provide an explicit justification and empirical evidence that sign-consistent solutions are enforced by the optimization.","section":"3.3 / A.3 (Eq. 18)"},{"comment":"The claim that GRAM is a reliable performance metric is weakened by a self-referential loop. The GRAM model is trained to minimize exactly the quantity that is then used as the metric, so a low GRAM value for this model is expected by construction. The correlation of ρ=0.923 is computed over three models, one of which (the GRAM model) is optimized for the metric; this does not establish that GRAM predicts performance on arbitrary multimodal models. I recommend recomputing the correlation on a set of models that were not trained with the GRAM objective, and reporting the correlation separately for models trained with and without GRAM. Without this, Section 3.5's claim that GRAM 'can serve as a metric for evaluating large multimodal models' is not supported.","section":"3.5 / Fig. 3"},{"comment":"The main comparisons against the VAST baseline are not controlled. The GRAM model is initialized from VAST pretrained weights and then further pretrained for one epoch on a 150k-sample subset of VAST27M with the GRAM loss, while the VAST row receives no such additional pretraining. Therefore the reported gains of +4.9 to +7.4 R@1 in Tables 1-2 and the gains in Table 3 cannot be attributed solely to the GRAM loss. The from-scratch ablation in Table 6 (Appendix B.2), which compares the GRAM loss with cosine TV-TA losses under identical training, is the correct controlled experiment and should be moved to the main text or at least carefully discussed alongside the headline numbers.","section":"Tables 1-3 / Section 4.1"}],"minor_comments":[{"comment":"For two vectors the volume is the area of a parallelogram, not a triangle; the sentence 'the volume computation degenerates to the area of the triangle' should be corrected.","section":"Section 4.4"},{"comment":"In the k > n case, the sentence 'the volume of the k-parallelotope is still positive but equal to zero' is contradictory; it should read 'is zero'.","section":"Section 3.2"},{"comment":"In the k < n case, the proof text says 'where k ≥ n'; this should be 'where k < n'.","section":"Appendix A.1"},{"comment":"The superscript/subscript notation in Eqs. (5)-(6) is not defined consistently: the text says m_x^y refers to the embedding of the x-th modality of the j-th sample, but the equations use m_i^j and m_j^k without a clear statement of which index is the sample and which is the modality.","section":"Section 3.4"},{"comment":"The reproducibility statement says pretrained models 'will be released after reviewing process,' while the abstract and introduction state they are available; please update for consistency.","section":"Reproducibility statement"},{"comment":"Minor typos: 'k-dimesnional' in Section 3.4, 'donwstream' in Section 4.4, and 'GRAMIAN' in the Appendix A title.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The sign-invariance issue is the central technical problem and, in my view, should be resolved before publication. The from-scratch ablation in Table 6 is the strongest positive evidence for the method, but it does not address the semantic meaning of the measure. If the authors can modify the measure to break sign symmetry and re-run the experiments, the paper could be suitable; otherwise the core claim fails."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely new idea — replace pairwise cosine similarity with the determinant of the Gram matrix, i.e., the volume of the parallelotope spanned by n modality vectors, as a multi-way alignment score. The math is standard but cleanly applied, the loss is simple, and the paper ships code, pretrained models, and a from-scratch ablation that isolates the loss from the pretraining confound. Worth a serious referee. But there are two soft spots that need airing.\n\nThe first is the sign-invariance problem, and it is real. Their own Appendix A.3 shows that for two unit vectors, Vol = |sin θ|, because the Gram determinant is unchanged by flipping any one vector (det(DGD) = det(G)). The loss in Eqs. 5–6 uses exp(-Vol/τ) as the alignment score, so an antiparallel negative pair is scored identically to an aligned positive pair. The paper never mentions this, and it directly contradicts the claim that \"lower volume means closer vectors.\" It also makes Section 3.5's use of GRAM as a model-performance metric questionable: a model with one modality globally sign-flipped would get a perfect GRAM score while being anti-aligned in the cosine sense the paper says it replaces. In practice, gradient descent from a sensible initialization may avoid the flipped basin, so the empirical retrieval numbers can still be valid. But the measure itself, as a measure of alignment, is weaker than advertised.\n\nThe second soft spot is the main comparison. The paper takes V AST pretrained models, adds one extra epoch of pretraining on 150k samples with the GRAM loss, and compares against the original, off-the-shelf V AST. That is not a controlled comparison; part of the 4.9–7.4 point gain could come from the extra training, not the loss. The from-scratch ablation in Table 6 is much fairer — it trains both the pairwise cosine loss and the GRAM loss from scratch with the same budget, and GRAM wins. That is the strongest evidence in the paper, and it does support the core claim. But the abstract's headline numbers come from the confounded setting.\n\nThe circularity concern is minor: the model is trained with the volume loss and then evaluated with a volume-based metric in Section 3.5, but the downstream R@1 numbers are independent, so this is not load-bearing.\n\nBottom line: the idea is promising and the from-scratch results suggest it has real value, but the sign invariance is a genuine geometric flaw that the authors need to address — either by modifying the measure (oriented volume, or adding a cosine term for the two-modality case) or by discussing why it does not matter empirically. The pretraining confound needs a matched baseline. I would send this to peer review, not desk reject, and I would ask the reviewers to push on both issues.","headline":"Neat Gramian-volume alignment idea with promising from-scratch results, but sign invariance and a pretraining confound keep the headline claims from fully landing.","tokens_in":23418,"tokens_out":2931,"would_cite":false,"duration_ms":29845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that aligning any number of modalities reduces to shrinking the parallelotope their normalized embeddings span, and that training models to do so reaches state-of-the-art video-audio-text retrieval and audio-video…","keywords":["Gramian matrix","parallelotope volume","multimodal alignment","contrastive learning","video-audio-text retrieval","cosine similarity generalization","modality gap","representation learning"],"falsifier":"Take a small two-modality retrieval benchmark, and for each query build a hard negative by flipping its true matching embedding to the antipode ($v \\mapsto -v$), so cosine similarity is $-1$ while the Gramian volume $\\sqrt{1 - \\cos^2\\theta} = 0$ is identical to the true positive's. If GRAM ranks these anti-aligned negatives as perfect matches, the claim that volume measures semantic alignment fails for that configuration; one could then look for this pattern arising in the trained model's own negatives.","tokens_in":22366,"feed_emoji":"📐","tokens_out":14517,"duration_ms":113343,"temperature":0.7,"pith_summary":"The paper's aim is to replace pairwise, anchor-based multimodal alignment with a single joint measure. GRAM computes, for any 2 to $n$ normalized modality embeddings, the volume of the $k$-dimensional parallelotope they span — the square root of the determinant of their Gram matrix — and treats smaller volume as better semantic alignment. The authors build a contrastive loss on this volume, minimizing it for matched samples, and train a video-audio-text model that reports state-of-the-art Recall@1 on retrieval benchmarks such as MSR-VTT, DiDeMo, ActivityNet, and VATEX, plus gains on audio tasks, improving on its cosine-based counterpart by 4.9 to 7.4 points. The paper also shows that the same volume correlates with downstream performance ($\\rho = 0.923$) and proposes GRAM as a metric for latent alignment. If correct, any existing multimodal model could swap cosine similarity for GRAM and align all modalities at once, with no architectural change.","feed_headline":"Modality alignment becomes one number: the volume they span","feed_subtitle":"One geometric volume replaces cosine similarity to align 2 to n modalities, lifting retrieval recall up to 7.4 points.","key_machinery":"The central object is the Gramian volume: for $k$ unit-norm embedding vectors, the square root of the determinant of their $k \\times k$ Gram matrix, $\\mathrm{Vol}(v_1,\\ldots,v_k) = \\sqrt{\\det G}$, which by a classical theorem is the volume of the $k$-dimensional parallelotope spanned by the vectors. The paper's machinery consists of using this single number both as the alignment measure and as the core of a contrastive loss, so one optimization objective couples all pairwise inner products at once. In the two-modality case the formula reduces to $\\sin\\theta$, a direct geometric generalization of cosine similarity, and for $k > 2$ it folds every pairwise term into one scalar, which is why the authors can claim simultaneous alignment of all modalities without a designated anchor.","core_discovery":"GRAM — the Gramian Representation Alignment Measure — is a similarity score defined across all modalities at once rather than pairwise. Given $k$ unit-norm modality embedding vectors $v_1,\\ldots,v_k \\in \\mathbb{R}^n$, the Gram matrix $G$ has entries $\\langle v_i, v_j \\rangle$, and the volume of the $k$-dimensional parallelotope they span is $\\mathrm{Vol} = \\sqrt{\\det G}$. The paper's central claim is that this volume is a faithful measure of joint semantic alignment: small volume means the modalities point in nearly the same direction in the shared embedding space, whereas cosine-based methods constrain only each modality against an anchor and leave non-anchor pairs unaligned. From this the authors derive a volume-based contrastive loss ($L_{D2A}$, $L_{A2D}$, plus a data-anchor matching loss $L_{DAM}$) that pulls matched modalities together and pushes negatives apart, and they prove the construction extends from $k=2$ up to $k=n$ modalities. Starting from the VAST backbone and pretraining briefly on a 150k-sample subset of VAST27M, the GRAM model reports state-of-the-art or improved Recall@1 on MSR-VTT, DiDeMo, ActivityNet, VATEX, AudioCaps, and VGGSound, with gains of 4.9 to 7.4 points over the cosine-based baseline in retrieval, and the paper further shows that $1 - \\mathrm{Vol}$ correlates with model performance ($\\rho = 0.923$), proposing GRAM as a metric for latent alignment.","pith_inferences":["The volume measure is sign-invariant: for two unit vectors it equals $\\sin\\theta$, so perfectly aligned ($\\theta=0$) and exactly anti-aligned ($\\theta=\\pi$) vectors get the same zero score, which means a negative pair lying in opposite directions would be scored as a perfect match by the GRAM loss in Eqs. (5)–(6) — a case the paper does not discuss.","The modality-gap measurements in the paper's Table 7 (larger inter-modality centroid distances after GRAM training) may be the flip side of that invariance: the loss can satisfy a small volume by pushing modality clusters to opposite sides of the hypersphere rather than by merging them, and neither the volume value nor the reported gap would reveal which happened.","A direct way to isolate the mechanism would be to train the same backbone with a cosine-based three-way objective that minimizes each modality's distance to the mean embedding of the others; if GRAM's gains persist, the advantage comes from joint alignment itself, not from the specific volume formulation.","Because the construction is defined for any $k \\le n$, GRAM could in principle align non-sensory embedding sets — multiple languages, multiple views, or heterogeneous feature groups — as long as the sign-invariance caveat is handled, though the paper only demonstrates sensory modalities."],"forward_implications":["Any downstream method that currently uses cosine similarity can swap in GRAM and align 2 to $n$ modalities jointly, without architectural modifications or added parameters.","In video-audio-text retrieval, the GRAM-trained model improves Recall@1 by 4.9 to 7.4 points over its cosine-based counterpart across zero-shot and fine-tuning settings on MSR-VTT, DiDeMo, ActivityNet, and VATEX.","Adding modalities one at a time — audio, then subtitles, then depth — raises zero-shot text-to-video Recall@1 on MSR-VTT from 52.8 to 55.3, supporting the claim that joint higher-dimensional alignment captures richer semantics than pairwise anchoring.","GRAM also functions as a performance metric: across LanguageBind, VAST, and the GRAM model on MSR-VTT, the rescaled $1-\\mathrm{Vol}$ score correlates with downstream Recall@1 at $\\rho = 0.923$."],"supporting_citations":[{"why":"Introduces the cosine-similarity contrastive loss and the two-modal alignment paradigm that GRAM generalizes to k > 2 modalities.","marker":"Radford et al. (2021)"},{"why":"Supplies the classical theorem that the square root of the Gram determinant equals the volume of the parallelotope, on which GRAM is built.","marker":"Gantmacher (1959)"},{"why":"Provides the VAST backbone encoders, the VAST27M pretraining subset, the lambda = 0.1 weighting, and the cosine-based baseline results GRAM is compared against.","marker":"Chen et al. (2023c)"},{"why":"Supplies the hard-negative mining strategy used for the data-anchor matching loss LDAM.","marker":"Li et al. (2021)"},{"why":"Defines the modality-gap measurements (centroid cosine distances) that Table 7 uses to analyze the GRAM latent space.","marker":"Liang et al. (2022)"},{"why":"ImageBind, an anchor-based (image-as-bridge) multimodal alignment baseline that GRAM claims to surpass.","marker":"Girdhar et al. (2023)"},{"why":"LanguageBind, a text-anchored n-modality alignment baseline whose embeddings and results GRAM is compared with.","marker":"Zhu et al. (2024)"}],"fun_headline_variants":["A single volume aligns all modalities at once","Gram volume replaces cosine for n-way alignment","One parallelotope volume aligns all embeddings","Modalities align when their Gram volume shrinks","Volume of spanned space = alignment score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on treating the volume spanned by the embeddings as a faithful measure of semantic alignment, but that equivalence is false for opposite-facing vectors: two unit vectors at angle $\\pi$ have the same zero volume as two perfectly aligned ones, so the score cannot distinguish 'aligned' from 'anti-aligned'.","fun_headline_variants_meta":{"raw":{"variants":["A single volume aligns all modalities at once","Gram volume replaces cosine for n-way alignment","One parallelotope volume aligns all embeddings","Modalities align when their Gram volume shrinks","Volume of spanned space = alignment score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2482,"prompt_tokens":1126,"completion_tokens":1356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":742,"completion_tokens_details":{"reasoning_tokens":1290}},"tokens_in":742,"tokens_out":1356,"duration_ms":13578,"temperature":1.0,"reasoning_tokens":1290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:25:09.023061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small two-modality retrieval benchmark, and for each query build a hard negative by flipping its true matching embedding to the antipode ($v \\mapsto -v$), so cosine similarity is $-1$ while the Gramian volume $\\sqrt{1 - \\cos^2\\theta} = 0$ is identical to the true positive's. If GRAM ranks these anti-aligned negatives as perfect matches, the claim that volume measures semantic alignment fails for that configuration; one could then look for this pattern arising in the trained model's own negatives.","supporting_citations":[{"cited_title":",vk) = p det G(v1,","cited_arxiv_id":null,"evidence_quote":"Supplies the classical theorem that the square root of the Gram determinant equals the volume of the parallelotope, on which GRAM is built."}],"review_version":1}