REVIEW 3 major objections 3 minor 2 references
VGGSounder: Audio-Visual Evaluations for Foundation Models
T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper argues that VGGSound misleads audio-visual evaluation and offers VGGSounder, a re-annotated multi-label benchmark, plus a metric that measures modality interference.
desk verdict The submitted full text is a different paper (ODYSSEY, a quadruped robotics paper), so VGGSounder's actual benchmark, annotation methodology, and metric cannot be evaluated from this artifact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are VGGSounder, a re-annotated multi-label extension of VGGSound, and the modality confusion metric. VGGSounder supplies per-clip ground truth that separates auditory labels from visual labels, so a model's audio-only, video-only, and audio-visual behavior can be compared against the same underlying content. The modality confusion metric measures the performance delta when another input modality is added; a negative delta signals that the extra modality actively misleads the model. Together they turn the benchmark from a single scalar accuracy number into a diagnostic for which modality carries the signal and which one interferes.
What would settle it
An independent annotation study that re-labels a random sample of VGGSounder clips without access to the original VGGSound labels, then measures agreement with VGGSounder's labels, would settle whether the ground truth is reliable. A second check: if the modality confusion metric is meaningful, models ought to show a similar ordering of audio-only, video-only, and audio-visual accuracy when evaluated on a small human-verified subset where the modality annotations are checked clip by clip.
Extended reading notes
Core claim
The paper's central claim is that VGGSound evaluation scores are distorted by three fixable defects: incomplete labels, partially overlapping classes, and modalities that do not actually line up in the recordings. VGGSounder addresses each defect with a re-annotated, multi-label test set that extends VGGSound and records modality-specific ground truth, so a clip can be labeled separately for its sound content and its visual content. Using this test set, the paper reports that existing audio-visual models often perform worse when a second modality is supplied, a phenomenon it quantifies with the modality confusion metric. The discovery, stated on the paper's own terms, is that a cleaner benchmark changes what evaluation reveals: apparent audio-visual competence partly dissolves, and modality interference becomes visible.
Load-bearing premise
The load-bearing premise is that VGGSounder's re-annotated labels and modality annotations are correct ground truth; if the annotation process is biased, subjective, or depends on the very foundation models under evaluation, the new benchmark would inherit the distortions it claims to fix.
Editorial extensions
If this is right
- VGGSound-based leaderboards that rank audio-visual models by top-1 accuracy may need revisiting, because a portion of the score differences reflects label and alignment artifacts rather than model competence.
- Model developers can use the modality confusion metric to detect when fusing audio and video degrades performance, and can target fusion layers or training data to fix it.
- The re-annotated multi-label format lets evaluators report separate audio and visual accuracies, so applications that care about one modality get a more precise comparison.
- A model that looks strong on VGGSound may appear weaker on VGGSounder; conversely, models trained with explicit modality supervision may be rewarded more fairly.
Reading between the lines
- A natural next step, not pursued in the abstract, is to release the annotation instructions and inter-annotator agreement; without those, the benchmark's objectivity cannot be audited.
- The modality confusion metric could be extended beyond classification to retrieval or generation, where 'adding a modality' might mean conditioning a text-to-audio model on video frames; the same negative-delta diagnostic would apply.
- If VGGSounder's labels are accepted, it would imply that part of the apparent progress in audio-visual classification over the last few years is an artifact of benchmark noise, and that future comparisons should control for modality alignment.
- The benchmark could be used to test whether contrastive audio-visual pretraining actually learns modality-specific features, by measuring whether fusion improves or degrades per-modality performance.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The submission, listed as arXiv:2508.08237 (cs.MM), presents an abstract claiming the introduction of VGGSounder, a comprehensively re-annotated, multi-label test set extending VGGSound for the evaluation of audio-visual foundation models, including detailed modality annotations and a new 'modality confusion metric.' The abstract further asserts that an analysis identifies limitations in VGGSound—incomplete labelling, partially overlapping classes, and misaligned modalities—that distort evaluations. However, the full text supplied with the submission is a completely different paper: 'ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Tasks' (arXiv:2508.08240, cs.RO), which concerns legged mobile manipulation and contains no mention of VGGSound, audio-visual classification, or the proposed metric. No methodology, annotation protocol, experimental results, or metric definitions for VGGSounder appear anywhere in the submission. The central claims are therefore unverifiable from the provided material.
Significance. If substantiated, the proposed contribution would be of real value to the audio-visual learning community: VGGSound is a widely used evaluation resource, and documented label problems together with a metric for modality interference would meaningfully strengthen benchmarking practice. Credit is due for stating a concrete problem and for recognizing that VGGSound's label noise and modality alignment issues can distort audio-visual evaluations. However, because the submission contains only the abstract and an unrelated robotics paper, no methodological contribution, reproducible artifact, or quantitative evidence can be assessed; the significance of the work cannot be established beyond the claims made in the abstract.
major comments (3)
- [Full text (entire submission)] The body of the submission is arXiv:2508.08240 ('ODYSSEY'), a quadruped-robot manipulation paper with no connection to VGGSound, audio-visual evaluation, or the proposed VGGSounder benchmark. This is the single load-bearing problem: the abstract's central claims—a limitations analysis of VGGSound, a re-annotation protocol, modality annotations, and a modality confusion metric—have no supporting content in the submission, so the soundness, statistical validity, and reproducibility of the proposed benchmark cannot be reviewed.
- [Abstract] The claimed analysis of VGGSound's limitations (incomplete labelling, partially overlapping classes, misaligned modalities) is asserted without any supporting data, examples, or described method; the manuscript must show how these limitations were identified and quantified before the motivation for VGGSounder can be accepted.
- [Abstract] The re-annotation methodology is not described anywhere, so the reader cannot rule out circularity: if the same foundation models under evaluation were used to generate the new labels or modality annotations, the benchmark would not fix the distortions it claims to address. The full manuscript must specify the annotation procedure, label schema, inter-annotator agreement, and independence from the models being evaluated.
minor comments (3)
- [Abstract] The phrase 'for evaluation audio-visual classification' contains a grammatical error and should read 'for evaluating audio-visual classification.'
- [Abstract] The submission should cite the original VGGSound dataset paper and provide pointers to the released annotations, code, and evaluation scripts, as is standard for benchmark papers.
- [Full text] The entire supplied PDF content (Figures 1, 9–11 and Table 8) pertains to the ODYSSEY robotics framework; this content should be replaced with the VGGSounder manuscript so that sections, equations, and tables can actually be located during review.
Circularity Check
No circularity can be established: the submitted full text is an unrelated paper (ODYSSEY, arXiv:2508.08240), so VGGSounder's derivation chain is absent from the reviewable material.
full rationale
The abstract describes VGGSounder, a re-annotated audio-visual test set with modality annotations and a modality confusion metric, but the accompanying full text is 'ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Tasks', a robotics paper with no audio-visual benchmark content. Consequently, there are no equations, annotation protocols, label-construction details, or metric definitions for VGGSounder to inspect. A circularity finding requires quoting the paper and exhibiting a specific reduction, such as a fitted parameter renamed as a prediction or a self-citation chain that forces the conclusion. No such reduction can be quoted because the relevant content is not present. The mismatch between abstract and full text is a serious submission-level integrity and completeness issue, not an argument-level circularity. It would be speculative to assign a circularity score on the basis of what the abstract does not describe; the absence of evidence is not evidence of circularity. Therefore the honest finding is no significant circularity (score 0).
Assumptions & free parameters
assumptions (3)
- domain assumption VGGSound is a representative and widely used audio-visual benchmark.
- ad hoc to paper The identified limitations (incomplete labelling, partially overlapping classes, misaligned modalities) are real and material.
- ad hoc to paper The re-annotation labels and modality annotations in VGGSounder are correct ground truth.
Cite this review
Pith. "Pith review of VGGSounder: Audio-Visual Evaluations for Foundation Models." pith.science (2026). https://pith.science/paper/UEFX2XFQ
@misc{pith2026250808237,
author = {Pith},
title = {Pith review of: VGGSounder: Audio-Visual Evaluations for Foundation Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/UEFX2XFQ}},
note = {Machine review of arXiv:2508.08237}
}
read the original abstract
The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding. The VGGSound dataset is commonly used as a benchmark for evaluation audio-visual classification. However, our analysis identifies several limitations of VGGSound, including incomplete labelling, partially overlapping classes, and misaligned modalities. These lead to distorted evaluations of auditory and visual capabilities. To address these limitations, we introduce VGGSounder, a comprehensively re-annotated, multi-label test set that extends VGGSound and is specifically designed to evaluate audio-visual foundation models. VGGSounder features detailed modality annotations, enabling precise analyses of modality-specific performance. Furthermore, we reveal model limitations by analysing performance degradation when adding another input modality with our new modality confusion metric.
Reference graph
Works this paper leans on
-
[1]
We present ODYSSEY , a unified mobile manipulation framework for agile quadruped robots equipped with manipulators, which seamlessly integrates high-level task planning with low-level whole-body control. Abstract Language-guided long-horizon mobile manipulation has long been a grand challenge in embodied semantic reasoning, gen- eralizable manipulation, a...
arXiv 2025
-
[2023]
and sophisticated manipulation techniques for object interaction in controlled settings (Kim et al. 2024; Bro- han et al. 2022; Cheang et al. 2024). While pioneering works (Pan et al. 2025a; Fu, Cheng, and Pathak 2023; Liu et al. 2024a; Zhang et al. 2025; Wang et al. 2024b; Fu, Zhao, and Finn 2024; Jiang et al. 2025b) have developed initial whole-body con...
work page 2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.