Pith. sign in

REVIEW 2 major objections 5 minor 59 references

SceneBind claims that adding object-centric semantic-spatial slots to a global scene embedding improves cross-modal scene retrieval, spatial retrieval, and zero-shot audio-visual localization over global-semantics-only embeddings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:41 UTC pith:5S4CVV7Z

load-bearing objection Solid system paper with a genuinely useful object-centric spatial layer, but the headline audio-spatial claims rest on Gemini-derived labels that the paper itself admits are not binaurally grounded. the 2 major comments →

arxiv 2607.15265 v1 pith:5S4CVV7Z submitted 2026-07-16 cs.CV cs.AIcs.MMcs.SD

SceneBind: Binding What and Where Across Vision, Audio and Language

Mingfei Chen , Zijun Cui , Ruoke Zhang , Hyeonggon Ryu , Eli Shlizerman This is my paper
classification cs.CV cs.AIcs.MMcs.SD
keywords omni-modal representationsemantic-spatial slotsbinaural audiocross-modal retrievalobject groundingaudio-visual localizationspatial retrievalzero-shot transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SceneBind claims that a scene is best represented not by a single global embedding but by a global embedding together with a small set of object-centric 'semantic-spatial slots,' each carrying what an object is and where it is (azimuth, elevation, distance) with a confidence. The paper shows this representation can be trained on top of frozen pretrained vision, audio, and text encoders with a few lightweight modules, supervised by bipartite matching to object-level text clauses and by contrastive alignment. On its curated real-world binaural benchmark, this what-and-where binding beats global-semantics-only embeddings on cross-modal scene retrieval, spatial retrieval, and object grounding, and transfers zero-shot to egocentric audio-visual localization. The upshot, if true, is that spatial structure is not a separate task but a cheap, learnable layer on existing semantic representations.

Core claim

On the paper's own terms, the discovery is that object-centric semantic-spatial slots are what carry the 'where' signal that global embeddings miss. The ablation isolates the causal role: adding slot matching raises spatial retrieval from 25.4 to 38.4 and text-conditioned scene retrieval from 34.2 to 39.9, while removing object semantic supervision drops grounding accuracy from 19.6 to 8.8. SceneBind trains these slots by bipartite matching predicted slots to ground-truth object clauses, combining global InfoNCE alignment, Gaussian-smoothed cross-entropy over discretized azimuth/elevation/distance bins, and intra-scene plus cross-scene slot contrastive losses. With about 10–25 slots (audio b

What carries the argument

The load-bearing object is the semantic-spatial slot, a tuple (s_k, r_k, c_k) where s_k is a semantic embedding decoded by a cross-attention module, r_k are predicted azimuth, elevation, and distance distributions over discretized bins, and c_k is an activity confidence. The spatial decoder uses each semantic slot as a query to attend to visual patch tokens or fused binaural audio context (semantic tokens plus a four-channel magnitude/phase spatial encoding). At inference, SceneBind Matching gates slots by confidence, matches each query slot to its best candidate slot under a semantic similarity times spatial-distribution-agreement score weighted by geometric-mean confidence, and adds this o

Load-bearing premise

The load-bearing premise is that the spatial labels used to supervise and evaluate the model are reliable; in particular, audio-only events receive direction and distance labels from a vision-language annotator that does not directly perceive binaural acoustic cues, so a systematic bias toward visible or front-center events would inflate the reported audio spatial competence.

What would settle it

Re-evaluate audio-only object grounding on a held-out set whose spatial labels are produced by a physically calibrated binaural measurement or a head-tracked human listener; if per-attribute audio accuracy collapses to near chance on those labels, the claim that audio slots learn genuine spatial structure is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Scene retrieval across audio–image, audio–text, and image–text improves when object-level spatial agreement is combined with global semantic similarity, with the largest gains on text-conditioned queries.
  • Semantically identical scenes that differ only in spatial layout (e.g., 360-degree viewpoint variants) become separable, which global embeddings cannot do.
  • Text queries can ground objects to per-attribute spatial predictions from either audio or vision, with audio contributing strong elevation cues and vision stronger azimuth/distance cues.
  • A low-cost spatial layer on frozen pretrained encoders suffices for state-of-the-art retrieval and zero-shot transfer, so this design can sit on top of any semantic encoder.
  • Zero-shot egocentric audio-visual localization benefits directly from the slot representation, outperforming prior finetuned approaches without task-specific training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the slot formulation is right, extending slots along time should let them carry motion trajectories instead of static positions, directly addressing the paper's noted limitation of short windows.
  • A testable next step is physically calibrating the audio spatial labels; the paper's own annotation caveat suggests audio accuracy may currently be vision-inflated, and cleaner labels could change the audio–vision balance.
  • The slot structure with confidence could serve as a grounded query language for spatial LLM reasoning: a language model that consumes global embedding plus slots gets an explicit coordinate system for 'where is X' questions.
  • One stress test implied by the method: measure retrieval performance as a function of object overlap between query and candidate scenes; best-match slot scoring should degrade more gracefully than global-only matching when scenes share no objects.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces SceneBind, an omni-modal scene representation that combines a global semantic embedding with K object-centric semantic-spatial slots, each carrying semantic features, predicted azimuth/elevation/distance, and a confidence score. The representation is extracted from frozen SigLIP2 and M2D-CLAP encoders, with lightweight trainable spatial decoders and matching modules. The authors curate a binaural audio-visual dataset with Gemini-generated semantic-spatial annotations, propose a bipartite-matching training objective, and evaluate cross-modal retrieval, spatial retrieval, object grounding, and zero-shot egocentric audio-visual localization. The central claims are that object-centric slots add spatial structure beyond global semantic embeddings and that SceneBind achieves state-of-the-art cross-modal scene and spatial retrieval while transferring zero-shot.

Significance. If the results hold, SceneBind is a useful and relatively inexpensive way to add object-level spatial grounding to existing frozen encoders, and the consistent ablations (Tables 3-5) support the causal role of object slots and the training objectives. The per-attribute grounding accuracies in Table 2 (e.g., 83.2% audio elevation, 66.1% visual distance) are far above the 0.7% all-attributes chance level and suggest the model learns nontrivial spatial associations. The two-stage training recipe and the zero-shot AV-localization results are also positive contributions. However, the audio-specific claims are weakened by the fact that the audio spatial ground-truth labels are generated by Gemini from visual evidence rather than from binaural cues, as the paper itself acknowledges in Appendix G. Because audio spatial performance is a headline result and a large fraction of the benchmark queries are audio-relevant, this data-loop concern must be resolved before the audio component of the contribution can be considered established.

major comments (2)
  1. [Appendix B.2 / B.3 / G; Sec. 3.3] The audio spatial supervision and evaluation labels come from the Gemini annotation pipeline, which Appendix G states 'does not directly perceive binaural spatial cues' and whose 'spatial estimates are inferred mainly from visual evidence.' For audio_only events (Fig. 8) there is by definition no visible object, so their azimuth/elevation/distance labels are LLM guesses conditioned on audio and visual context rather than on inter-channel binaural information. These same labels supervise the audio spatial decoder through the object grounding loss in Sec. 3.3 and define the ground truth for audio-relevant queries: 745 of 1,527 spatial-retrieval queries are audio-relevant (B.4), and Table 2 reports audio elevation accuracy 83.2%. A systematic bias in Gemini's guesses -- e.g., a front/center prior or a tendency to align with visible objects -- would inflate the reported audio spatial compete
  2. [Sec. 3.2 / Sec. 3.3; Table 2] The paper claims that audio spatial understanding comes from binaural cues, and it adds a four-channel time-frequency spatial encoder to the monaural semantic audio tokens (Sec. 3.2). Yet there is no ablation that removes the binaural spatial encoder or replaces the binaural input with mono audio. Given that the training labels are themselves vision-derived (Appendix G), the audio spatial decoder might be learning to reproduce Gemini's semantic/positional priors rather than exploiting inter-channel level and phase differences. The ablations in Tables 3-5 vary matching and losses but never vary the audio input evidence. An experiment comparing SceneBind's audio branch with and without the binaural spatial features is needed to support the specific claim that the representation captures true spatial audio cues.
minor comments (5)
  1. [Table 6] The column layout is hard to parse: the header 'Audio–Text Visual–Text A V VT AT Avg' mixes task names and metric abbreviations. Please separate the three retrieval directions and the metric columns explicitly.
  2. [Sec. 1 / Sec. 3.2] The abstract says 'a few additional tokens,' but the model uses K=50 object queries per modality (Appendix C.1). Clarify that the overhead is small relative to the 256 visual patch tokens, not that the absolute number of tokens is small.
  3. [Appendix B.3] The human review process is described qualitatively, but no inter-reviewer agreement or correction-rate statistics are reported. Even a brief table of reviewer decisions (keep/discard/correct) would help calibrate the reliability of the benchmark labels.
  4. [Sec. 4.4 / Table 1] The sentence 'achieving 65.3 on V↔T (+28% over pretrained, +48% over finetuned)' appears to mix percentage-point and relative-improvement language. Please state the comparison metric explicitly and consistently.
  5. [Appendix G] The main text should cross-reference Appendix G's limitation that Gemini does not perceive binaural cues. As written, the main text presents the audio spatial results without acknowledging this significant caveat.

Circularity Check

1 steps flagged

Audio spatial claims are entangled with Gemini's vision-derived labels; the core slot representation and matching scheme are otherwise self-contained.

specific steps
  1. other [Sec. B.2 (Annotation), Sec. 3.3 (spatial supervision), Sec. 4.2/G (evaluation and limitations)]
    "Although Gemini does not directly perceive binaural spatial cues, it provides strong visual recognition... estimate the corresponding camera-coordinate spatial labels from the visual evidence. /// Gemini ... does not directly perceive binaural spatial cues. Its spatial estimates are inferred mainly from visual evidence, which can bias annotations toward visible or front-facing objects and may miss or hallucinate off-screen or rear-field events."

    The audio spatial decoder is trained (Sec. 3.3: L_theta, L_phi, L_d) and evaluated (Table 2, Sec. 4.2) on the same Gemini-generated spatial labels that, by the paper's own admission, were inferred from the visual stream rather than from binaural cues. For audio_only events there is no visible object, so their labels are LLM guesses. Reported audio spatial accuracy (e.g., 83.2% elevation) therefore measures fit to a vision-derived label-generation prior, not independent binaural perception. This is a partial circularity confined to the audio spatial claim; the slot-matching equations and external AVLoc zero-shot evaluation are not circular.

full rationale

The central derivation — global embedding + K semantic-spatial slots (Eq. 1), bipartite matching supervision (Eq. 3), and SceneBind Matching (Eq. 6) — is not a re-derivation of its own output. Tables 3 and 4 are internal ablations, and Table 7 evaluates zero-shot transfer on an external egocentric audio-visual localization benchmark, which provides independent evidence that the representation carries spatial signal. There is no load-bearing uniqueness theorem or ansatz smuggled in via self-citation; the authors' prior work appears only as background context. The one genuine circularity is the audio spatial label loop: the benchmark and training labels for audio events originate from Gemini, which the paper itself states cannot perceive binaural cues and infers spatial estimates mainly from visual evidence, biasing toward visible/front-facing objects. The same labels supervise and score the audio spatial decoder, so the reported audio spatial accuracy is partially a measure of how well the model reproduces the label-generation prior rather than true binaural spatial understanding. Human review of the benchmark does not dissolve this loop because it starts from the same Gemini proposals and reliability for audio_only spatial corrections is not quantified. This warrants a moderate score, not a high one, because the visual/text slot machinery and the external zero-shot evaluation remain independent and non-circular.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The central claim rests on roughly eight hand-set hyperparameters and four domain assumptions about the reliability of LLM-generated spatial labels and pretrained-encoder alignment; no new physical entities are introduced. The object-centric semantic-spatial slot is a representational construct (DETR-style), not a claimed new constituent of the world.

free parameters (8)
  • Number of object slots K = 50
    Chosen by hand; Fig. 3 shows performance saturates beyond ~10 slots and audio peaks near 5, so 50 is conservative but no sensitivity analysis is reported.
  • Inference weighting λobj = 0.05 (A–V), 0.5 (text-related)
    Test-time weighting of object score vs global score (Sec. C.4), tuned separately per retrieval direction on the benchmark.
  • Bipartite matching coefficients λ, α, β (Eq. 3) = not reported
    Matching score mixes semantic and spatial terms; values are not given in main text or appendix, leaving assignment supervision incompletely specified.
  • Loss weights (Table 9) = 15 weights, e.g., 2.0 for L_av_global and L_conf, 1.0/0.5/0.5 for azimuth/elevation/distance
    Full objective is a weighted sum; no sensitivity analysis or search procedure is reported.
  • Confidence gate and reranking window = 0.05; top-50
    Active-slot threshold and global-prefilter size chosen by hand (Sec. C.4).
  • Spatial smoothing σ = 1.0 azimuth (circular), 0.5 elevation/distance
    Gaussian-smoothed cross-entropy bandwidths chosen without stated justification (Sec. C.2).
  • Verification thresholds and balancing keep rate = ImageBind ≥0.10, CLAP ≥0.05; 50% keep for front-center
    Event-filtering thresholds (Table 8) and spatial-balancing rate (Sec. B.2) shape the dataset and are chosen ad hoc.
  • AV-localization guide-map temperatures = 0.45, 0.60, 0.75; σx=σy=0.13
    Zero-shot localization pipeline parameters (Sec. E) tuned for the downstream cIoU metric.
axioms (6)
  • domain assumption Binaural waveforms contain recoverable azimuth/elevation/distance cues that a from-scratch conv frontend can extract under frozen mono-trained semantic tokens.
    Sec. 3.2/C.1: spatial encoder trained on 4-channel STFT features; the paper does not quantify binaural-cue SNR in its corpus.
  • domain assumption M2D-CLAP audio patch tokens can be aligned into SigLIP2's vision-text space with a residual MLP and safely frozen afterwards.
    Sec. 3.2/C.1: alignment MLP pretrained on Binaural captions; assumes the two pretrained spaces share sufficient structure.
  • domain assumption Gemini's visual-evidence-based spatial estimates are accurate enough to serve as supervision for both visual and audio spatial attributes.
    Fig. 8 prompt and Sec. B.2; the paper's own Sec. G concedes this is biased toward visible/front-facing objects.
  • domain assumption ImageBind and CLAP verification scores (thresholds ≥0.10/≥0.05) suffice to filter LLM-hallucinated events and correctly downgrade modality types.
    Table 8; no analysis of verification error rates is provided.
  • standard math In-batch InfoNCE negatives provide unbiased cross-modal alignment across heterogeneous sources (Binaural, AudioCaps, MS-COCO).
    Sec. C.2; standard contrastive-learning assumption, not validated for dataset-mixing bias.
  • standard math Hungarian bipartite matching yields the correct supervision assignment despite noisy predicted slots and variable object counts.
    Sec. 3.3, Eq. 3; optimal assignment is standard, but the matching score depends on the untested λ, α, β.

pith-pipeline@v1.3.0-alltime-deepseek · 26152 in / 18936 out tokens · 170159 ms · 2026-08-01T23:41:45.438781+00:00 · methodology

0 comments
read the original abstract

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.

Figures

Figures reproduced from arXiv: 2607.15265 by Eli Shlizerman, Hyeonggon Ryu, Mingfei Chen, Ruoke Zhang, Zijun Cui.

Figure 1
Figure 1. Figure 1: SceneBind models scenes by jointly capturing semantics ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SceneBind overview. (a) SceneBind Encoding Model maps multimodal inputs to a global embedding s global and object-centric slots (sk, rk, ck). (b) SceneBind Representation X represents each scene with a global embedding and K object-centric semantic-spatial slots. (c) Bipartite matching aligns slots with GT object clauses for supervision. (d) SceneBind Matching combines global similarity Sglobal with object… view at source ↗
Figure 3
Figure 3. Figure 3: Object slot scaling. Per￾formance saturates or drops beyond a moderate number; ∼10 slots suffice [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results. SceneBind retrieves semantically and spatially aligned scenes across audio, vision, and text. We show top-3 candidates with global similarity (g), object-centric matching score (o), and grounded slots with confidence and spatial labels. 4.7 Zero-shot Downstream Application: Egocentric Spatial Audio-Visual Localization SceneBind generalizes to egocentric AV localization without task-spe… view at source ↗
Figure 5
Figure 5. Figure 5: AV localization visualization. Global semantic attention proposes sounding regions, refined by spatial consistency of SceneBind slots. to model object motion, long-range scene dynamics, and temporally consistent semantic-spatial reasoning. Additional discussion is provided in the appendix. 6 Acknowledgment The authors (MC,ES) acknowledge the partial support of HDR Institute: Accelerated AI Algorithms for D… view at source ↗
Figure 6
Figure 6. Figure 6: Long-tail distributions of scene tags and semantic event clusters in the Binaural dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Binaural data construction funnel. The pipeline filters source videos, selects diverse [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Audio–Visual event and spatial annotation prompt for binaural videos via Gemini. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Azimuth distribution of the primary audio–visual event per clip, before ( [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Human verification interface for reviewing candidate benchmark clips. It allows reviewers [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: A→V beach retrieval. SceneBind ranks the ground truth first by matching both the global beach semantics and the object level spatial cue of waves, while baseline retrievals capture coarse beach semantics but may miss the spatial layout. Case 2: Multiple Slots Capture Moving Sound Sources [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: T→V river retrieval. SceneBind matches both the global river road context and object level spatial cues of the riverbed and vehicles, while ImageBind is less sensitive to the detailed text query and M2D-SigLIP2* misses key objects or retrieves weaker layouts [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: A→V moving bus retrieval. SceneBind uses multiple bus engine relevant slots to represent plausible semantic spatial hypotheses for a moving sound source, retrieving scenes with road and vehicle layouts that better match the binaural audio. how SceneBind slots expose object-level spatial structure while also revealing uncertainty for moving sources. Case 3: Audio as the Retrieved Target [PITH_FULL_IMAGE:f… view at source ↗
Figure 14
Figure 14. Figure 14: T→A jazz retrieval. SceneBind ranks the ground-truth audio first, and its top retrieved samples show similar jazz performance context, spectrogram and waveform patterns, and object level spatial cues for instruments such as saxophone and drums [PITH_FULL_IMAGE:figures/full_fig_p030_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: V→A street retrieval. SceneBind retrieves audio with similar street-scene acoustics and frontal vehicle cues. Although the ground truth is ranked second, the top results remain semantically and spatially close, illustrating the challenge of scene retrieval when audio is the target modality. G Limitations and Future Works Data Scale and Annotation Quality. SceneBind relies on curated spatially aligned audi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 10 linked inside Pith

  1. [1]

    Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    SA VVY: Spatial awareness via audio-visual LLMs through seeing and hearing

    Mingfei Chen, Zijun Cui, Xiulong Liu, Jinlin Xiang, Caleb Zheng, Jingyuan Li, and Eli Shlizerman. SA VVY: Spatial awareness via audio-visual LLMs through seeing and hearing. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  3. [3]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  4. [4]

    Sigmoid loss for language image pre-training.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023

    2023

  5. [5]

    Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H’enaff, Jeremiah Harmsen, Andreas Steiner, and Xiao-Qi Zhai

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim M. Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier H’enaff, Jeremiah Harmsen, Andreas Steiner, and Xiao-Qi Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense feature...

    Pith/arXiv arXiv 2025

  6. [6]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  7. [7]

    M2d-clap: Masked modeling duo meets clap for learning general-purpose audio-language representation, 2024

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Masahiro Yasuda, Shun- suke Tsubaki, and Keisuke Imoto. M2d-clap: Masked modeling duo meets clap for learning general-purpose audio-language representation, 2024

    2024

  8. [8]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023

    2023

  9. [9]

    Soundspaces 2.0: A simulation platform for visual-acoustic learning.Advances in Neural Information Processing Systems, 35:8896– 8911, 2022

    Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning.Advances in Neural Information Processing Systems, 35:8896– 8911, 2022

    2022

  10. [10]

    Spatial-clap: Learning spatially-aware audio–text embeddings for multi-source conditions.arXiv preprint arXiv:2509.14785, 2025

    Kentaro Seki, Yuki Okamoto, Kouei Yamaoka, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. Spatial-clap: Learning spatially-aware audio–text embeddings for multi-source conditions.arXiv preprint arXiv:2509.14785, 2025

    arXiv 2025

  11. [11]

    Audioclip: Extending clip to image, text and audio, 2021

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio, 2021

    2021

  12. [12]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  14. [14]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning, 2023

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning, 2023. 11

    2023

  15. [15]

    Robohop: Segment-based topological map representation for open-world visual navigation

    Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for open-world visual navigation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4090–4097. IEEE, 2024

    2024

  16. [16]

    Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion: Open-set multimodal 3d mapping.Robotics: Science and Systems (RSS), 2023

    2023

  17. [17]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024

    2024

  18. [18]

    3d-mem: 3d scene memory for embodied exploration and reasoning, 2024

    Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning, 2024

    2024

  19. [19]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. InProceedings of the IEEE/CVF International conference on computer vision, pages 15625–15636, 2023

    2023

  20. [20]

    Visual language maps for robot navigation

    Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023

    2023

  21. [21]

    Chatsplat: 3d conversational gaussian splatting

    Hanlin Chen, Fangyin Wei, and Gim Hee Lee. Chatsplat: 3d conversational gaussian splatting. arXiv preprint arXiv:2412.00734, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Language embedded 3d gaus- sians for open-vocabulary scene understanding

    Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open-vocabulary scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333–5343, 2024

    2024

  23. [23]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    2024

  24. [24]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.ArXiv, abs/2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.ArXiv, abs/2303.05499, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Grounding dino 1.5: Advance the "edge" of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding dino 1.5: Advance the "edge" of open-set object detection. ArXiv, abs/2405.10300, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors,Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, Lecture Note...

    2020

  27. [27]

    Deformable DETR: deformable transformers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021

    2021

  28. [28]

    Visual scene graphs for audio source separation.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1184–1193, 2021

    Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio source separation.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1184–1193, 2021. 12

    2021

  29. [29]

    Learning audio-visual dynamics using scene graphs for audio source separation

    Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. Learning audio-visual dynamics using scene graphs for audio source separation. InAdvances in Neural Information Processing Systems, 2022

    2022

  30. [30]

    Audio-visual grouping network for sound localization from mixtures

    Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10565–10574, 2023

    2023

  31. [31]

    Lavss: Location-guided audio-visual spatial audio separation

    Yuxin Ye, Wenming Yang, and Yapeng Tian. Lavss: Location-guided audio-visual spatial audio separation. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5508–5519, 2024

    2024

  32. [32]

    Audio-visual scene analysis with self-supervised multisen- sory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisen- sory features. InProceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018

    2018

  33. [33]

    Discriminative sounding objects localization via self-supervised audiovisual matching

    Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33:10077–10087, 2020

    2020

  34. [34]

    Cyclic co-learning of sounding object visual grounding and sound separation.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2744–2753, 2021

    Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2744–2753, 2021

    2021

  35. [35]

    Sound event localization and detection of overlapping sources using convolutional recurrent neural networks

    Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1):34–48, 2018

    2018

  36. [36]

    Robust sound source tracking using srp-phat and 3d convolutional neural networks.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:300–311, 2020

    David Diaz-Guerra, Antonio Miguel, and Jose R Beltran. Robust sound source tracking using srp-phat and 3d convolutional neural networks.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:300–311, 2020

    2020

  37. [37]

    Bat: Learning to reason about spatial sounds with large language models.International conference on machine learning, 2024

    Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learning to reason about spatial sounds with large language models.International conference on machine learning, 2024

    2024

  38. [38]

    Learn- ing spatially-aware language and audio embeddings

    Bhavika Suresh Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, YELENA MENYAYLENKO, Barry-John Theobald, Jonathan Sheaffer, and Miguel Sarabia. Learn- ing spatially-aware language and audio embeddings. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  39. [39]

    Phasecoder: Microphone geometry-agnostic spatial audio understanding for multimodal llms, 2026

    Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, and Vivek Kumar. Phasecoder: Microphone geometry-agnostic spatial audio understanding for multimodal llms, 2026

    2026

  40. [40]

    Hear you are: Teaching llms spatial reasoning with vision and spatial sound

    Hyeonggon Ryu, Joon Son Chung, and David Harwath. Hear you are: Teaching llms spatial reasoning with vision and spatial sound. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 38606–38615, 2026

    2026

  41. [41]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Pith/arXiv arXiv 2018

  42. [42]

    Harold W. Kuhn. The hungarian method for the assignment problem.Naval Research Logistics (NRL), 52, 1955

    1955

  43. [43]

    Audiocaps: Generat- ing captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. InNAACL-HLT, 2019

    2019

  44. [44]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014. 13

    2014

  45. [45]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Omniaudio: Generating spatial audio from 360-degree video.ArXiv, abs/2504.14906, 2025

    Huadai Liu, Tianyi Luo, Qikai Jiang, Kaicheng Luo, Peiwen Sun, Jialei Wan, Rongjie Huang, Qian Chen, Wen Wang, Xiangtai Li, Shiliang Zhang, Zhijie Yan, Zhou Zhao, and Wei Xue. Omniaudio: Generating spatial audio from 360-degree video.ArXiv, abs/2504.14906, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Egocentric audio-visual object localization

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22910–22921, 2023

    2023

  48. [48]

    Learning to localize sound source in visual scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018

    2018

  49. [49]

    Space-time memory network for sounding object localization in videos.arXiv preprint arXiv:2111.05526, 2021

    Sizhe Li, Yapeng Tian, and Chenliang Xu. Space-time memory network for sounding object localization in videos.arXiv preprint arXiv:2111.05526, 2021

    Pith/arXiv arXiv 2021

  50. [50]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16867–16876, 2021

    2021

  51. [51]

    Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes

    Zengjie Song, Yuxi Wang, Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang. Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3222–3231, 2022

    2022

  52. [52]

    Mix and localize: Localizing sound sources in mixtures

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483–10492, 2022

    2022

  53. [53]

    Seeing speech and sound: Distinguishing and locating audio sources in visual scenes

    Hyeonggon Ryu, Seongyu Kim, Joon Son Chung, and Arda Senocak. Seeing speech and sound: Distinguishing and locating audio sources in visual scenes. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13540–13549, 2025

    2025

  54. [54]

    Cnn architectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131–135. IEEE, 2017

    2017

  55. [55]

    Hrtf measurements of a kemar dummy-head microphone

    Bill Gardner and Keith Martin. Hrtf measurements of a kemar dummy-head microphone. 1994. 14 A Appendix Overview In this appendix, we provide supplementary details and analyses that complement the main paper. • Dataset Curation(Sec. B): We describe the data sources, annotation pipeline, verification process, balancing strategy, and benchmark statistics. • ...

    1994

  56. [56]

    - visual_only: if visible, but it is silent or not synchronized with any sound

    Event Modality Constraints - audio_only: if you can hear the object, but it is not in the field of view. - visual_only: if visible, but it is silent or not synchronized with any sound. - audio_visual: if visible AND the sound is clearly synchronized with its motion AND there is strong audio-visual correspondence. High precision required

  57. [57]

    - Duration Constraint: Events must be short atomic instances

    Annotation Rules (Strict) - Timestamps: Must be relative to clip start (0s to 10s). - Duration Constraint: Events must be short atomic instances. Duration must be EXACTLY 1 or 2 seconds. - Correct Format: [1,3), or [2,3) relative to clip start; Incorrect Format: [00:01:21-00:01:23) absolute video timestamps. - Split Rule: If an event lasts longer than 2 s...

  58. [58]

    Spatial Definitions (Camera Coordinates) Azimuth (Horizontal): Center is 0 deg; Negative is left; Positive is right. - Hard Left: [-90, -60) - Left: [-60, -30) - Slight Left: [-30, -15) - Front: [-15, 15] - Slight Right: (15, 30] - Right: (30, 60] - Hard Right: (60, 90] Elevation (Vertical): Level is 0 deg; Negative is Down; Positive is Up. - High Above: ...

  59. [59]

    - semantic_anno (6 to 8 words): Describe WHAT the object is doing/being

    Field Descriptions - semantic_tag: A concise semantic tag for the event, e.g., ‘car honking’, ‘playing piano’. - semantic_anno (6 to 8 words): Describe WHAT the object is doing/being. Strictly Forbidden: Directional words. - spatial_anno (about 8 words): Describe WHERE the object is. Strictly Forbidden: Naming the object, complex action. - combined_anno (...