Pith. sign in

REVIEW 4 major objections 6 minor 72 references

SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces SAVVY-Bench and shows that a training-free pipeline lifts dynamic audio-visual 3D QA from 50.9% to 58.0%.

desk verdict A genuinely new dynamic audio-visual 3D spatial reasoning benchmark with a clever training-free pipeline, but the audio distance contribution is partly circular and the benchmark's answer key needs independent validation before the gains can be fully trusted. read the letter →

arxiv 2506.05414 v1 pith:MWS7BDZG submitted 2025-06-04 cs.CV cs.AIcs.LGcs.MMcs.SDeess.AS

Mingfei Chen , Zijun Cui , Xiulong Liu , Jinlin Xiang , Caleb Zheng , Jingyuan Li , Eli Shlizerman This is my paper
classification cs.CVcs.AIcs.LGcs.MMcs.SDeess.AS
keywords audio-visualLLMs3Dspatialreasoningegocentricreferenceframeallocentricaudiodynamicscenesbenchmarkglobalmap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SAVVY-Bench is a new benchmark that asks audio-visual language models to answer 3D spatial questions about dynamic scenes with synchronized spatial audio, covering egocentric and allocentric direction and distance across thousands of QA pairs. The paper's main claim is that existing AV-LLMs, even the strongest proprietary ones, perform poorly on these questions because they ignore spatial audio and cannot track a sounding object through time in a consistent world frame. To fix this, the authors propose SAVVY, a training-free pipeline that estimates per-frame egocentric tracks for the objects in a question, merges them into a global map via SLAM camera trajectories, and then reads off the answer from that map. On the benchmark, SAVVY raises overall QA accuracy from 50.9% (Gemini-2.5-pro) to 58.0%, a +7.1% gain, with especially large gains on allocentric direction questions. If the results hold, the practical point is that strong spatial reasoning can be bolted onto existing AV-LLMs without retraining, and that spatial audio plus explicit geometry is the missing ingredient.

What carries the argument

The load-bearing object is the egocentric spatial track, a per-frame tuple $(t, \theta, r)$ giving the timestamp, azimuth relative to the camera ($0^\circ$ forward, negative left, positive right), and distance in meters of an object. The second stage folds these tracks into a dynamic global map $M_q = \{p_{\mathrm{sound}}(t) \mid t \in T_q\} \cup \{p_{\mathrm{ref}}, p_{\mathrm{face}}\}$ by adding the SLAM camera trajectory $L(t)$ to each track: $p(t) = L(t) + (r\cos\theta,\, r\sin\theta)$. The map is what converts egocentric observations into allocentric answers: reference and facing objects become static landmarks defining a rotated coordinate frame, while the sounding object remains a time-varying trajectory. The identity doing the work is the coordinate transformation—once everything lives in one global frame, direction and distance questions reduce to reading off the map, which is also why missing tracks can be filled in by whichever modality (vision, audio, or language) observes the object at that moment.

What would settle it

Re-annotate a random subset of SAVVY-Bench with independent ground truth, for example lidar-measured object centers and manually time-stamped sound events, and re-run the model comparisons; if the 58.0% versus 50.9% gap narrows or flips, the original answer key, not spatial reasoning, was driving the results.

Watch

Extended reading notes

Core claim

The central discovery is that dynamic 3D spatial reasoning does not require a new trained model: it can be decomposed into egocentric track estimation and global map construction. Given a question and a video with multi-channel audio, SAVVY first uses an audio-visual LLM to write a sparse snapshot description of the query-referenced event, identifying the target, reference, and facing objects with time spans and rough positions. It then fills in trajectory gaps with text-guided segmentation (CLIPSeg/SAM2 plus metric depth) and with spatial audio cues (SRP-PHAT for direction, coherent-to-diffuse ratio for distance). All per-frame estimates are lifted into a single global 2D map using the SLAM camera trajectory, static objects are clustered, and a Kalman filter smooths the dynamic target's path. The answer is obtained by a coordinate transformation aligned to the queried viewpoint, which is why the same pipeline handles both egocentric and allocentric questions. This construction yields the reported 58.0% overall accuracy, up from 50.9% for Gemini-2.5-pro, and 78.6% sounding-object localization accuracy when all three track types are combined.

Load-bearing premise

The benchmark's ground-truth positions, event times, and source labels must be accurate—object centers are automated detections that annotators corrected in a point-cloud viewer, speech topics come from a prompted model, and sound events are human-labeled—because if the answer key is systematically wrong, every model score, including the reported +7.1 gain, measures errors in the key rather than spatial reasoning.

Editorial extensions

If this is right

  • Augmenting an AV-LLM with a training-free spatial pipeline raises overall benchmark accuracy from 50.9% to 58.0%, with the largest relative gain on allocentric direction questions (+12.3%).
  • Spatial audio is the decisive cue when the sounding object leaves the camera view: audio-only tracking reaches egocentric direction accuracy comparable to Gemini-2.5-pro, and the full pipeline corrects errors that arise from extrapolating the last visible location.
  • Combining all three track types (snapshot descriptor, segmentation, audio) gives the best sounding-object localization (78.6% loc_acc) and the best egocentric QA, while the snapshot descriptor is what makes allocentric reference objects reliable.
  • Humans score 78.7% against the best model's 58.0%, and 7B-scale AV-LLMs are near chance on allocentric direction and below 5% temporal-mIoU, indicating that event-time grounding and object referral, not spatial math alone, are the current bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the +7.1 point gain is driven by the global-map stage rather than any single sensor module, then the same two-stage decomposition could be compressed into a training objective for AV-LLMs; the paper names this as future work, but the ablation tables make it a testable prediction.
  • The paper's constant-$K$ approximation $D_t^2 \cdot \mathrm{CDR}_t \approx K$ predicts that audio-based distance estimates degrade in rooms with changing reverberation; measuring CDR stability across the recorded environments would reveal whether audio distance or visual depth is the weaker link.
  • Because the speech topics in the benchmark are produced by a prompted language model from transcripts, part of the measured difficulty may be topic paraphrase rather than spatial reasoning; re-running the QA templates with independently human-generated speech topics would isolate that factor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SAVVY-Bench, a new benchmark for 3D spatial question answering in dynamic, audio-visual scenes, built from the Aria Everyday Activities dataset and comprising egocentric and allocentric direction and distance questions. It also proposes SAVVY, a training-free pipeline that combines an AV-LLM 'Snapshot Descriptor', CLIPSeg/SAM2 segmentation with metric depth, SRP-PHAT direction-of-arrival estimation, CDR-based distance estimation, and SLAM-derived global mapping. The main empirical claim is that SAVVY raises overall QA accuracy from 50.9% (Gemini-2.5-pro) to 58.0% on this benchmark, with ablations attributing gains to the three egocentric track components.

Significance. The benchmark is the first of its kind for dynamic 3D audio-visual spatial reasoning and could become a useful community resource if the ground-truth pipeline is validated and the data are released. The paper is strong in its detailed ablations, its use of real egocentric data with 7-channel spatial audio, the inclusion of a human baseline, and the qualitative error analysis. The central claims, however, rest on two points that need work: the CDR distance calibration is partly circular, and the benchmark answer-key validation is not reported. Both issues are fixable within the scope of a revision, but they affect the credibility of the headline +7.1% improvement and the claimed complementary role of spatial audio for distance estimation.

major comments (4)
  1. [Section 4.2] The CDR-based distance estimate is calibrated using the same visual distance estimates D_t that the pipeline is later said to complement. The text defines d_hat_t = sqrt(K/CDR_t) with K = argmin_K sum_t (D_t^2 · [CDR_t - K])^2. As written the optimization is dimensionally inconsistent: the product D_t^2·CDR_t does not appear in the objective. If the intended expression is sum_t (D_t^2·CDR_t - K)^2, then K is fitted to the visual D_t on the same video, and the audio distance estimate reduces to a rescaled copy of the visual estimate. Consequently, the 'Audio' track in Table 6 is not an audio-only distance estimator, and the claim that spatial audio improves distance accuracy is overstated. Please fix the equation, calibrate K on held-out scenes or with a fixed per-environment constant that does not use the test video's visual D_t, and re-evaluate the audio-only ablation.
  2. [Section 3.2 / C.2] The benchmark answer key is the load-bearing element for every comparison in Table 2, but its reliability is not quantified. Object positions are EFM3D detections manually corrected in a point-cloud viewer, speech topics are generated by prompting Gemini on word-level transcripts, and sound-event times and source identities are labeled by human annotators; no inter-annotator agreement, correction statistics, or independent 3D validation is reported. Because direction labels flip on quadrant boundaries and distance accuracy is judged at 1 m tolerance, systematic errors in source positions or event times would shift all model scores and could absorb or inflate part of the reported +7.1% gain. Please report agreement statistics and validate a sample of 3D positions against an independent reconstruction or multiple annotators, and release the benchmark artifacts so that the community can check this risk.
  3. [Section 5.3 / Table 6] The claimed 'Audio-only' performance is not achieved with audio alone for distance. Since K is fitted from the SD/Seg visual distance estimates on the same video, the audio distance component inherits the visual estimates, so the 48.1% egocentric distance entry in Table 6 does not measure audio-only distance estimation. An analysis that separates the SRP-PHAT direction (which is vision-free) from the CDR distance (which currently is not) is needed to support the paper's conclusion that 'spatial audio cues complement visual input for robust tracking'.
  4. [Section 3.1 / release] The paper states that SAVVY-Bench comprises 'thousands of QA pairs' but does not report the total number of QA pairs, videos, or sound events, and the project website does not currently provide the benchmark artifacts or code. For a benchmark paper, this is a load-bearing reproducibility issue: without a dataset release or an explicit release plan, the community cannot verify the ground-truth quality or use the benchmark. Please include a data card with exact counts, a release link, and a commitment to make the evaluation pipeline available.
minor comments (6)
  1. [Figure 3(c)] The figure shows d_t = K * CDR, which conflicts with the text's d_hat_t = sqrt(K/CDR_t); please make the notation consistent.
  2. [Section 4.2] In the K optimization expression, the placement of brackets makes the intended objective ambiguous; write the objective explicitly as sum_t (D_t^2 * CDR_t - K)^2 or state the intended expression in words.
  3. [Table 2] Please specify how the 'overall' accuracy is computed, in particular whether it is a weighted average over the four task types or an unweighted average.
  4. [Figure 2] Please add the exact counts or percentages on the pie slices and label the axes of the density plots; the current figure does not convey the number of questions behind each distribution.
  5. [Section 5.1] The distance metric called 'average relative accuracy' is defined as the average across absolute error thresholds; this name is easy to confuse with relative error, so please clarify that it is a thresholded absolute-error accuracy.
  6. [Section 3.2 vs C.2] The main text says a proprietary AV-LLM is used to extract word-level transcriptions, while Section C.2 says the AEA raw annotations already contain word-level transcriptions and the annotators cluster them; please clarify which source is used to avoid an apparent inconsistency.

Circularity Check

1 steps flagged · score 4.0 of 10

One pipeline component—the CDR-based audio distance estimate—is fitted to the same visual distance estimates it is meant to supplement; the rest of the SAVVY comparison is self-contained.

  1. fitted input called prediction [Section 4.2, 'Spatial Audio Cues' (CDR distance formula)]
    "To estimate distance, we adopt the coherent-to-diffuse ratio (CDR) approach [67]. We compute CDR at each time frame and use distance estimates from the visual-guided modules (Snapshot Descriptor and text-guided snapshot segmentation) to exploit the acoustic property that D2 t ·CDR t remains approximately constant in a given environment. We estimate this constant K by computing D2 t ·CDR t per frame t, applying DBSCAN [68] to filter outliers, and minimizing the squared error over remaining frames: ˆdt = q K CDRt , where K= arg min K P t D2 t · [CDRt −K]2."

    K is fitted from D_t, which are the visual distance estimates produced by the Snapshot Descriptor and text-guided segmentation modules. The 'audio' distance d̂_t = sqrt(K / CDR_t) is therefore a transformation of those visual estimates rather than an independent spatial-audio measurement. When Table 6 reports Audio-only distance accuracy (48.1) or the combined pipeline's distance gains, that distance signal has been calibrated to the very visual tracks it is supposed to complement, so part of the claimed audio contribution reduces to the visual estimates by construction. The SRP-PHAT direction channel and the SLAM-based global mapping remain independent, which is why this is a partial rather than total circularity.

full rationale

The central SAVVY-vs-Gemini comparison is not itself derived from its inputs: the pipeline adds SRP-PHAT direction estimation, CLIPSeg/SAM2 text-guided segmentation, metric depth, SLAM-based global mapping, and explicit coordinate transformations, all of which are external, stated-assumption components with their own ablations. The paper's self-citations ([55], [72]) are not load-bearing for the main claims. The benchmark ground-truth construction (EFM3D detections corrected by annotators, Gemini-generated speech topics, human-labeled sound events) is a data-quality risk but not a circular derivation, because the answer key is external to the model predictions being scored. The one genuine construction-level circularity is the CDR distance calibration: the paper explicitly fits K using the visual distance estimates, then reports the resulting quantity as an audio distance cue. That makes the 'Audio' distance component in Tables 5, 6, and 9 partly a re-expression of the SD/Seg visual distances, so the distance-related contribution of spatial audio is not fully independent. Directional audio cues, however, are computed independently via SRP-PHAT, and the global-mapping gains are separately demonstrated with and without the audio track. Overall, the paper has partial circularity in one component but retains substantial independent content.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central evaluation rests on the accuracy of ground-truth annotations, the reliability of the audio model's assumptions, and the correctness of the SLAM/depth tools. None of these are machine-checked or shipped. The most consequential free parameter is K, the audio-distance scale fitted to visual distance estimates.

free parameters (4)
  • K (CDR distance scale) = per scene, fitted via least squares
    Used to convert CDR into audio distance: K = argmin sum D_t^2(CDR_t-K)^2 with D_t from visual tracks; makes audio distance partially dependent on visual estimates (Section 4.2).
  • CLIPSeg confidence thresholds = 0.5 (sounding), 0.6 (reference/facing)
    Chosen by ablation in Table 7; affect detection validity and downstream map quality.
  • Number of sampled frames N_frame = 128
    Ablated in Table 8; higher N improves sounding object localization but can hurt allocentric QA.
  • Audio fusion frustum and forward rejection = ±1 m, 45°, 10x5 bins; [-5°,5°] forward rejected
    Hand-chosen parameters in the audio track refinement (Section F.3) that affect final trajectory.
assumptions (6)
  • domain assumption D_t^2 * CDR_t remains approximately constant in a given environment
    Assumed in Section 4.2 to convert CDR into distance; fails with multiple sources or strong reverberation.
  • domain assumption SRP-PHAT on the Aria 7-mic array gives a reliable direction of arrival for the dominant sound source
    Used in Section 4.2 and Table 4; front/back accuracy is only 75.0% in best configuration.
  • domain assumption The AEA SLAM/VIO camera trajectory is accurate
    Used to transform egocentric tracks into the global map (Section 4.1, E.3); drift would corrupt the map.
  • domain assumption ZoeDepth provides accurate metric distances indoors
    Monocular metric depth used in text-guided snapshot segmentation (Section 4.2).
  • domain assumption The SAVVY-Bench ground truth (EFM3D detections plus manual refinement) is accurate
    All benchmark scores and the human baseline are computed against these annotations (Section 3.2).
  • domain assumption Gemini's temporal grounding and object referral are reliable enough
    SAVVY's Snapshot Descriptor stage depends on Gemini; the paper states this in Section H.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing." pith.science (2026). https://pith.science/paper/MWS7BDZG

@misc{pith2026250605414,
  author       = {Pith},
  title        = {Pith review of: SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MWS7BDZG}},
  note         = {Machine review of arXiv:2506.05414}
}
read the original abstract

3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLMs) and benchmarks, which predominantly focus on static or 2D scenes. We introduce SAVVY-Bench, the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of relationships involving static and moving objects, and requires fine-grained temporal grounding, consistent 3D localization, and multi-modal annotation. To tackle this challenge, we propose SAVVY, a novel training-free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs as well as other audio-visual methods to track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi-modal queried object trajectories and converts them into a unified global dynamic map. Using the constructed map, a final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state-of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.

Figures

Figures reproduced from arXiv: 2506.05414 by the authors.

Figure 1
Figure 1. 3D spatial reasoning in dynamic audio-visual environments. The task requires fine-grained [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. SAVVY-Bench comprises thousands of QA pairs that probe spatial relationships involving [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Benchmark Statistics. (a) Task distribution by type. (b) Angle distribution of queries over 360°. (c) Distribution of query distances. (d) Video duration distribution. 3 SAVVY-Bench 3.1 Overview SAVVY-Bench is the first benchmark for evaluating 3D spatial reasoning of AV-LLMs in dynamic, multi-room scenes. It builds on the egocentric Aria-Everyday Activities (AEA) dataset [60], which includes over 600 sound events a… view at source ↗
Figures from the paper (13 more)
Figure 3
Figure 3. Figure 3: SAVVY consists of two stages: Given a query and video with spatial audio, stage 1 extracts Egocentric Spatial Tracks with (a) “Snapshot” Descriptors via AV-LLMs, (b) Text-Guided Snapshot Segmentation, and (c) Spatial Audio Cues. Stage 2 constructs a dynamic Global Map …
Figure 4
Figure 4. Figure 4: Example reasoning process of AV-LLMs. Top (Egocentric direction); Bottom (Allocen￾tric distance). Middle (SAVVY): SAVVY successfully fixes the spatial relation errors. allocentric distance example in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Human-in-the-Loop Dataset Curation and Benchmark Construction Workflow for SAVVY [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Prompt used to generate speech topics from word-level transcripts. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Interface for sound event annotation. The tool displays dual-camera videos with synchronized playback and saves annotations locally. SAVVY-Bench includes six templates covering four task types: egocentric direction, egocentric distance, allocentric direction, and alloc…
Figure 8
Figure 8. Figure 8: Review interface for QA pair quality review. The tool displays each video clip alongside its associated question and predicted answer, allowing reviewers to efficiently assess correctness, clarity, and formatting, and make a decision on whether the QA pair is a good QA…
Figure 9
Figure 9. Figure 9: Direction quadrant guide for human evaluation. Egocentric directions are relative to the [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prompt for Open-Source AV-LLMs on SAVVY-Bench. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Prompt for Proprietary AV-LLMs (Gemini 2.5 models) on SAVVY-Bench. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Example of Referral Error. The model incorrectly selects the armchair (facing object) at the arched opening, failing to identify the correct object description queried in the question. This misinterpretation further causes error for the spatial relationship. Imagine y…
Figure 13
Figure 13. Figure 13: Example of Temporal Localization Error. The model incorrectly grounds the speech event “suggesting trying the coffee,” confusing it with a similar topic. This mismatch causes incorrect egocentric direction prediction. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Example of Spatial Relationship Error. The model correctly identifies the queried time span, relevant objects, and their locations, but fails to determine the correct relative direction of the sounding object with respect to the robot’s view. It incorrectly predicts t…
Figure 15
Figure 15. Figure 15: Example of Spatial Measurement Error. The model correctly identifies the sound source and reference objects, but fails to compute the distance accurately along with the navigation route from the reference object to the sound source. 28 [PITH_FULL_IMAGE:figures/full_f…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 22 canonical work pages

  1. [1]

    Shelton and Timothy P

    Amy L. Shelton and Timothy P. McNamara. Systems of spatial reference in human memory. Cognitive Psychology, 43(4):274–312, 2001

    work page 2001

  2. [2]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465, June 2024

    2024

  3. [3]

    Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    arXiv 2024

  4. [4]

    Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion: Open-set multimodal 3d mapping.Robotics: Science and Systems (RSS), 2023

    2023

  5. [5]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    arXiv 2024

  6. [6]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  7. [7]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16488–16498, 2024

    2024

  8. [8]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108–19118, 2022

    2022

Show all 72 references
  1. [9]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision a...

    2022

  2. [10]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  3. [11]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision).arXiv preprint arXiv:2309.17421, 9(1):1, 2023

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision).arXiv preprint arXiv:2309.17421, 9(1):1, 2023

    2023 arXiv

  4. [12]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  5. [13]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  6. [14]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  7. [15]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 29

    2023 arXiv

  8. [16]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  9. [17]

    Listen, think, and understand.arXiv preprint arXiv:2305.10790, 2023

    Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand.arXiv preprint arXiv:2305.10790, 2023

    2023 arXiv

  10. [18]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  11. [19]

    Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    2025 arXiv

  12. [20]

    Audio-reasoner: Improving reasoning capability in large audio language models, 2025

    Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-reasoner: Improving reasoning capability in large audio language models, 2025

    2025

  13. [21]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  14. [22]

    Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos.arXiv preprint arXiv:2411.19772, 2024

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos.arXiv preprint arXiv:2411.19772, 2024

    2024 arXiv

  15. [23]

    Egolife: Towards egocentric life assistant, 2025

    Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Ya...

    2025

  16. [24]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04328, 2025

    2025 arXiv

  17. [25]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  18. [26]

    video-SALMONN: Speech-enhanced audio-visual large language models

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, Yuxuan Wang, and Chao Zhang. video-SALMONN: Speech-enhanced audio-visual large language models. InForty-first International Conference on Machine Learning, 2024

    2024

  19. [27]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  20. [28]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning, 2023

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning, 2023

    2023

  21. [29]

    Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  22. [30]

    Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023. 30

    2023 arXiv

  23. [31]

    Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu- Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models.arXiv preprint arXiv:2404.07973, 2024

    2024 arXiv

  24. [32]

    Robohop: Segment-based topological map representation for open-world visual navigation

    Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for open-world visual navigation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4090...

    2024

  25. [33]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In2024 IEEE International Conference...

    2024

  26. [34]

    3d-mem: 3d scene memory for embodied exploration and reasoning, 2024

    Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning, 2024

    2024

  27. [35]

    3d-llava: Towards generalist 3d lmms with omni superpoint transformer.arXiv preprint arXiv:2501.01163, 2025

    Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub, and Ian Reid. 3d-llava: Towards generalist 3d lmms with omni superpoint transformer.arXiv preprint arXiv:2501.01163, 2025

    2025 arXiv

  28. [36]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding.arXiv preprint arXiv:2412.00493, 2024

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding.arXiv preprint arXiv:2412.00493, 2024

    2024 arXiv

  29. [37]

    Gridmm: Grid memory map for vision-and-language navigation

    Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. InProceedings of the IEEE/CVF International conference on computer vision, pages 15625–15636, 2023

    2023

  30. [38]

    Visual language maps for robot navigation

    Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023

    2023

  31. [39]

    Chatsplat: 3d conversational gaussian splatting

    Hanlin Chen, Fangyin Wei, and Gim Hee Lee. Chatsplat: 3d conversational gaussian splatting. arXiv preprint arXiv:2412.00734, 2024

    2024 arXiv

  32. [40]

    Language embedded 3d gaus- sians for open-vocabulary scene understanding

    Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open-vocabulary scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333–5343, 2024

    2024

  33. [41]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  34. [42]

    Sound event localization and detection of overlapping sources using convolutional recurrent neural networks

    Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13(1):34–48, 2018

    2018

  35. [43]

    Robust sound source tracking using srp-phat and 3d convolutional neural networks.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:300–311, 2020

    David Diaz-Guerra, Antonio Miguel, and Jose R Beltran. Robust sound source tracking using srp-phat and 3d convolutional neural networks.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:300–311, 2020

    2020

  36. [44]

    Bat: Learning to reason about spatial sounds with large language models.arXiv preprint arXiv:2402.01591, 2024

    Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learning to reason about spatial sounds with large language models.arXiv preprint arXiv:2402.01591, 2024

    2024 arXiv

  37. [45]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  38. [46]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  39. [47]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  40. [48]

    Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models.arXiv preprint arXiv:2311.16103, 2023

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models.arXiv preprint arXiv:2311.16103, 2023

    2023 arXiv

  41. [49]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  42. [50]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  43. [51]

    Egotaskqa: Understanding human tasks in egocentric videos.Advances in Neural Information Processing Systems, 35:3343–3360, 2022

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos.Advances in Neural Information Processing Systems, 35:3343–3360, 2022

    2022

  44. [52]

    Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models

    Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. InEuropean Conference on Computer Vision, pages 331–348. Springer, 2024

    2024

  45. [53]

    Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  46. [54]

    Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos.arXiv preprint arXiv:2503.13646, 2025

    Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos.arXiv preprint arXiv:2503.13646, 2025

    2025 arXiv

  47. [55]

    Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering

    Xiulong Liu, Zhikang Dong, and Peng Zhang. Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4478–4487, 2024

    2024

  48. [56]

    Avqa: A dataset for audio-visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. InProceedings of the 30th ACM international conference on multimedia, pages 3480–3491, 2022

    2022

  49. [57]

    Pano-avqa: Grounded audio-visual question answering on 360deg videos

    Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2031–2041, 2021

    2021

  50. [58]

    Egocentric audio-visual object localization

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22910–22921, 2023

    2023

  51. [59]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022

    2022

  52. [60]

    Aria everyday activities dataset.arXiv preprint arXiv:2402.13349, 2024

    Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria everyday activities dataset.arXiv preprint arXiv:2402.13349, 2024

    2024 arXiv

  53. [61]

    Efm3d: A benchmark for measuring progress towards 3d egocentric foundation models.arXiv preprint arXiv:2406.10224, 2024

    Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, and Richard Newcombe. Efm3d: A benchmark for measuring progress towards 3d egocentric foundation models.arXiv preprint arXiv:2406.10224, 2024. 32

    2024 arXiv

  54. [62]

    Photo-slam: Real-time simul- taneous localization and photorealistic mapping for monocular stereo and rgb-d cameras

    Huajian Huang, Longwei Li, Hui Cheng, and Sai-Kit Yeung. Photo-slam: Real-time simul- taneous localization and photorealistic mapping for monocular stereo and rgb-d cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21584–21593, 2024

    2024

  55. [63]

    Image segmentation using text and image prompts

    Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086–7096, 2022

    2022

  56. [64]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenh...

    2024 arXiv

  57. [65]

    Zoedepth: Zero-shot transfer by combining relative and metric depth.arXiv preprint arXiv:2302.12288, 2023

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth.arXiv preprint arXiv:2302.12288, 2023

    2023 arXiv

  58. [66]

    Brown University, 2000

    Joseph Hector DiBiase.A high-accuracy, low-latency technique for talker localization in reverberant environments using microphone arrays. Brown University, 2000

    2000

  59. [67]

    Coherent-to-diffuse power ratio estimation for dere- verberation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(6):1006– 1018, 2015

    Andreas Schwarz and Walter Kellermann. Coherent-to-diffuse power ratio estimation for dere- verberation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(6):1006– 1018, 2015

    2015

  60. [68]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. InProceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, page 226–231. AAAI Press, 1996

    1996

  61. [69]

    A new approach to linear filtering and prediction problems.Transactions of the ASME–Journal of Basic Engineering, 82(Series D):35–45, 1960

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems.Transactions of the ASME–Journal of Basic Engineering, 82(Series D):35–45, 1960

    1960

  62. [70]

    The pascal visual object classes (voc) challenge.International Journal of Computer Vision, 88:303–338, 06 2010

    Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge.International Journal of Computer Vision, 88:303–338, 06 2010

    2010

  63. [71]

    Lmms-eval: Reality check on the evaluation of large multimodal models.arXiv preprint arXiv:2407.12772, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models.arXiv preprint arXiv:2407.12772, 2024

    2024 arXiv

  64. [72]

    Xiulong Liu, Sudipta Paul, Moitreya Chatterjee, and Anoop Cherian. Caven: An embodied conversational agent for efficient audio-visual navigation in noisy environments.Proceedings of the AAAI Conference on Artificial Intelligence, 38(4):3765–3773, Mar. 2024. 33

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.