Pith. sign in

REVIEW 4 major objections 4 minor 73 references

Differentiable Room Acoustic Rendering with Multi-View Vision Priors

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining multi-view visual features with acoustic beam tracing in a differentiable renderer yields accurate room impulse response estimation from sparse real-world measurements, outperforming prior methods and matching models trained on…

desk verdict Genuinely novel integration of beam tracing with multi-view visual cues, but the vision-prior claim is only ablated in one room, so treat the multimodal headline as promising rather than proven. read the letter →

arxiv 2504.21847 v2 pith:2FXIWBDT submitted 2025-04-30 cs.CV cs.SD

Derong Jin , Ruohan Gao This is my paper · ORCID
classification cs.CVcs.SD
keywords roomimpulseresponsedifferentiablerenderingacousticbeamtracingaudio-visuallearningmulti-viewvisionmaterial-awarereflectionspatialaudioreal-worldRIRestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Realistic spatial audio requires knowing a room's impulse response—how sound bounces from a speaker to a listener—but dense measurements are costly and pure learning methods need huge datasets. The paper proposes that a room's appearance can supply most of the missing information, because visible surface materials largely determine how sound reflects. Its AV-DAR system feeds multi-view images through a vision encoder whose material-aware features condition a differentiable beam-tracing renderer, so reflection responses are learned from a handful of recorded RIRs. On two real-world datasets with six rooms, AV-DAR trained on 0.1% of the measurements matches baselines trained on 1%, and at equal scale it improves relative error by 16.6% to 50.9%. If the visual-acoustic correlation holds, this makes per-room acoustic capture cheap enough for consumer AR/VR.

What carries the argument

The central object is the differentiable RIR renderer built on acoustic beam tracing, with a vision-conditioned multi-scale reflection response. Beam tracing represents sound as volumetric cones rather than zero-width rays, so a listener is registered as 'hit' by a specular path without Monte Carlo oversampling; each path's frequency response is the product of reflection responses at hit points, mapped to the time domain by a minimum-phase transform and accumulated with propagation loss and delay. Because a beam's footprint on a surface is an ellipse whose size grows with travel distance, the reflection response is evaluated at an integrated positional encoding (IPE) that averages Fourier features over the elliptical region. A multi-view vision encoder supplies a material-aware feature at each surface point, aggregated across cameras by cross-attention and across neighboring samples by point-transformer fusion, so the same point can have different effective reflection responses depending on viewing context and beam scale. A residual neural field treats every surface point as a secondary source and Monte Carlo-integrates over solid angles to model diffuse reflections and late reverberation. This decomposition lets gradients flow from the RIR loss back through all components, fine-tuning the visual features themselves for acoustic prediction.

What would settle it

Take a room whose walls wear a single uniform paint hiding different substrates (drywall on one side, concrete masonry on another), record RIRs, and compare AV-DAR with and without the visual branch: if the vision-conditioned model no longer beats the acoustic-only version, the visual-prior premise is falsified for that setting. More directly, impedance-tube measurements of two surface samples with identical RGB appearance but different absorption coefficients would yield reflection responses that a vision-only encoder cannot distinguish.

Watch

Extended reading notes

Core claim

AV-DAR is the claim that RIR rendering can be made both physical and data-lean by letting vision supply the surface reflection properties that acoustics alone cannot identify from sparse measurements. The paper decomposes the RIR into a learnable source response, a reflection response computed by tracing specular beams through coarse room geometry, and a residual field capturing diffuse reflections and late reverberation. Vision enters through a multi-view encoder that aggregates pixel-aligned features across cameras into a material-aware descriptor at each surface point, conditioning the frequency-dependent reflection response at every beam hit. The entire pipeline is optimized end-to-end against ground-truth RIRs, and on the Real Acoustic Field dataset the model trained on 0.1% of data matches baselines trained on 1%, with relative gains of 16.6% to 50.9% at equal scale; on four Hearing Anything Anywhere rooms trained on 12 locations it outperforms all prior physics- and learning-based baselines on nearly every metric.

Load-bearing premise

The entire method rests on the assumption that surface appearance reliably predicts acoustic reflection behavior—if visually similar materials reflect sound very differently, or the same material looks different under varied lighting, the vision encoder will train on a misleading signal and the data-efficiency gains vanish.

Editorial extensions

If this is right

  • A new room can be captured with a camera and a handful of microphone recordings rather than tens of thousands of source–listener pairs.
  • Physics-based rendering becomes fast enough for interactive use: inference stays under 70 ms for a two-second RIR in the tested rooms, an order of magnitude faster than differentiable image-source rendering.
  • The learned acoustic model is inspectable: optimized reflection responses align with material categories (carpet absorbs high frequencies, metal reflects them), so engineers can trace where a prediction comes from.
  • The approach transfers across very different real venues—offices, a classroom, a hallway, a dampened room—indicating the physics-vision combination generalizes beyond the training scenes rather than overfitting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the visual-to-acoustic correlation holds across scenes, the same vision encoder could be trained across many rooms to enable zero-shot RIR prediction for never-seen spaces from images alone—an extension the paper explicitly leaves to future work.
  • The two-level cross-attention aggregation of multi-view features is a general mechanism: any inverse problem where image evidence constrains physical surface parameters (e.g., thermal emissivity, tactile roughness) could reuse this pattern of conditioning a differentiable physical simulator.
  • The current method relies on known rough geometry (e.g., a few planes); coupling AV-DAR's beam tracer with automatic image-based geometry estimation would remove the last manual step, yielding a purely vision-driven acoustic renderer.
  • Because the residual field is position-dependent and learned, the same decomposition could be reused to estimate physical parameters such as frequency-dependent absorption coefficients that other room acoustic solvers could consume.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces AV-DAR, a differentiable room acoustic rendering framework that combines acoustic beam tracing with multi-view visual features. The RIR is decomposed into a source response, a beam-traced reflection response conditioned on visual features and integrated positional encodings, and a learned residual field for late reverberation and diffraction. The method is evaluated on the Real Acoustic Field (RAF) dataset and the Hearing Anything Anywhere (HAA) dataset, reporting improved C50, EDT, T60, and loudness errors over learning-based and physics-based baselines, including a claimed 10x data-efficiency advantage on RAF and large same-scale gains. The paper also provides ablations, qualitative wave-field visualizations, interpretable reflection-response maps, and supplementary computational-cost and failure-case analyses.

Significance. If the claims hold, AV-DAR would be a meaningful step toward few-shot, physics-based room impulse response estimation, and the paper contains several strengths: it is, to my knowledge, the first to integrate acoustic beam tracing into a fully differentiable end-to-end RIR renderer; it evaluates on six real-world rooms from two datasets; it includes component ablations, view-count ablations, failure cases, and a disclosed data-quality exclusion in the supplement; and the learned reflection responses are visualized and shown to be material-aware. However, the headline claims about the multi-view vision prior and about 'significantly outperforming' baselines are not yet fully supported by the evidence as presented: all comparisons are single-run, the vision ablation is confined to one scene and one training scale, and a key metric in the 10x-data comparison is not favorable to the method. The central architecture is plausible and the empirical direction is promising, but the load-bearing evidence needs strengthening before the paper's strongest claims can be accepted.

major comments (4)
  1. [§4.2, Tables 1, 2, 3, and 6] All reported results are single runs without error bars, multiple random seeds, or significance tests. The abstract and §4.2 repeatedly state that AV-DAR 'significantly outperforms' prior methods, but no statistical support is provided. This matters because several differences are small or inconsistent; for example, in Table 3 the T60 error is lower without vision (14.3) than with the full model (15.2). Please report variance over multiple training runs and, where appropriate, paired significance tests for the main comparisons.
  2. [§4.2, Table 3 and ablation paragraph] The vision-prior ablation ('w/o Vision') is reported only for RAF-Furnished at 0.1% training data. Since the multi-view vision prior is the central contribution stated in the title and introduction, the absence of w/o Vision results for RAF-Empty, the four HAA rooms, and the 1% training scale is a load-bearing gap: the large reported gains could be driven primarily by the beam-tracing plus residual structure rather than by the visual features. The text's statement that 'each component is essential' is also not uniformly supported by Table 3, where removing vision improves T60. Please add cross-scene vision ablations and discuss metric-specific effects.
  3. [§4.1, Implementation Details (HAA images)] For the HAA dataset, the multi-view 'images' are rendered from Polycam reconstructions rather than real photographs. The paper discloses this, but the central claim concerns visual priors from multi-view images, and the HAA experiments therefore rest on an unvalidated assumption that rendered Polycam views are an acceptable substitute for real photographs. The manuscript should explicitly frame HAA as testing the method with rendered visual inputs, and either provide evidence that this surrogate preserves the relevant material cues or add a caveat that real-photo performance on HAA is not yet evaluated.
  4. [§4.2, Table 1 and '10x data' claim] The statement that AV-DAR 'achieves comparable performance to models trained on 10 times more data' is not uniformly true across the reported metrics. For example, on RAF-Furnished EDT, Ours at 0.1% data reports 80.1 ms, which is worse than AVR at 1% data (72.3 ms) and NAF++ at 1% data (74.9 ms). The claim should be qualified to identify the metrics and rooms for which the 10x-data comparison holds, and the paper should avoid implying overall superiority on all metrics.
minor comments (4)
  1. [§3.5, Eq. (15)] The word 'quires' appears in the text describing the attention mechanism; it should be 'queries.'
  2. [§A.2.4] There is a duplicated phrase 'the importance of importance of visual information' and a typo 'evluat- ing' in the supplementary; these should be corrected.
  3. [§4.1, Implementation Details] The paper specifies that HAA images are rendered at 512x512 resolution from Polycam reconstructions, but it does not state how many rendered views are used per HAA room; please provide these counts for reproducibility.
  4. [§3.3 and §A.1.4] The beam-tracing apex angle, the number of beams, the source directional sharpness parameter, and the frequency discretization are free parameters, but no sensitivity analysis is reported; a short sensitivity study or a justification for the chosen values would strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AV-DAR is a standard supervised differentiable renderer whose parameters are learned from held-out RIR measurements against external benchmarks; the vision prior is an input modality, not a target-derived construction.

full rationale

The derivation chain is self-contained as supervised inverse rendering. The forward model (Eq. 4) decomposes the RIR into a source response, a beam-traced specular reflection response conditioned on integrated positional encoding and vision features (Eqs. 10-13), and a Monte-Carlo residual field (Eqs. 20-22). All components are optimized against measured RIRs using the loss in Eq. 27, and evaluation uses held-out source-listener pairs in the RAF and HAA benchmarks. No predicted quantity is defined by the same measured values used to fit it; the '10x data' comparisons and 16.6%-50.9% gains are direct empirical comparisons on test splits. The paper does invoke prior work by its own authors—[67] for the HAA dataset, training splits, and the source directivity hyperparameter—but these are published datasets and modeling choices, not unverified theorems, and they do not carry the derivation. The supplementary material honestly reports limitations (e.g., an invalid RIR in the 0.0003% subset in Supp. A.2.2; view-count saturation where 'with only 10 views the model nearly abandons visual feature learning' in Supp. A.2.4; failure cases in Supp. A.2.6). These weaken the strength of the vision-prior contribution, especially since the w/o Vision ablation is reported only for RAF-Furnished at 0.1%, but they are empirical caveats, not circular reductions. No equation reduces to a fitted constant, and no prediction is defined in terms of the target quantity.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The paper's central claim rests on domain assumptions about geometric acoustics and visual-material correlation, plus several hand-chosen hyperparameters. No new physical entities are introduced. The learned residual is a flexible component that could absorb failures of the physics model, which is the main reason soundness is rated only 5.

free parameters (4)
  • beam tracing apex angle = not reported (described as 'small')
    Hand-chosen in Section 3.3 to keep beams disjoint; no value or sensitivity analysis is given, yet it directly controls which reflection paths are detected.
  • number of beam samples = 16,384 training / 8,192 inference
    Reported in Supp A.1.5; scales computation and path coverage; chosen by hand.
  • source directional response sharpness parameter = 8
    Taken from [67] (Supp A.1.4) for the spherical Gaussian weighting of D_x; not learned and not tuned on the target datasets.
  • frequency discretization for reflection responses = 16 log-spaced points, 12-7800 Hz
    Reported in Supp A.1.4; the reflection response is interpolated across these hand-selected key frequencies.
assumptions (7)
  • domain assumption Geometric acoustics with specular reflections approximates the early part of the room impulse response.
    Section 3.3 models only specular paths via beam tracing; diffraction and diffuse scattering are deferred to the learned residual (Section 3.6).
  • domain assumption Visual appearance correlates with acoustic surface properties.
    Section 1: 'a surface region's acoustic property often strongly correlates with visual appearance due to the same underlying materials'; this motivates the entire multi-view vision module.
  • domain assumption Narrow, non-splitting beams cover all significant specular paths.
    Section 3.3 states beam splitting is neglected and a small apex angle is chosen; if important paths are missed, the residual must compensate.
  • domain assumption The minimum-phase transform reconstructs a valid time-domain response from the magnitude-only product of reflection responses.
    Equation 7 applies MinPhase to the frequency-domain product; this assumes minimum-phase behavior for reflection paths.
  • domain assumption Accurate room geometry and camera poses are available.
    The method requires known camera intrinsics/extrinsics and planar or reconstructed geometry (Sections 3.1, 4.1); errors in this input would propagate into beam tracing and vision feature alignment.
  • ad hoc to paper Rendered images from Polycam reconstructions are an acceptable substitute for real multi-view photographs in HAA.
    Section 4.1: since HAA provides no images, the authors render 512x512 images from the Polycam reconstructions; this is a pragmatic adaptation specific to this paper.
  • domain assumption The learned residual can represent high-order reflections, diffuse reflections, diffraction, and late reverberation.
    Section 3.6 parameterizes the residual with a 4-layer MLP and Monte Carlo integration; the flexibility of this component determines how much unmodeled physics it can absorb.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentiable Room Acoustic Rendering with Multi-View Vision Priors." pith.science (2026). https://pith.science/paper/2FXIWBDT

@misc{pith2026250421847,
  author       = {Pith},
  title        = {Pith review of: Differentiable Room Acoustic Rendering with Multi-View Vision Priors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FXIWBDT}},
  note         = {Machine review of arXiv:2504.21847}
}
read the original abstract

An immersive acoustic experience enabled by spatial audio is just as crucial as the visual aspect in creating realistic virtual environments. However, existing methods for room impulse response estimation rely either on data-demanding learning-based models or computationally expensive physics-based modeling. In this work, we introduce Audio-Visual Differentiable Room Acoustic Rendering (AV-DAR), a framework that leverages visual cues extracted from multi-view images and acoustic beam tracing for physics-based room acoustic rendering. Experiments across six real-world environments from two datasets demonstrate that our multimodal, physics-based approach is efficient, interpretable, and accurate, significantly outperforming a series of prior methods. Notably, on the Real Acoustic Field dataset, AV-DAR achieves comparable performance to models trained on 10 times more data while delivering relative gains ranging from 16.6% to 50.9% when trained at the same scale.

Figures

Figures reproduced from arXiv: 2504.21847 by the authors.

Figure 1
Figure 1. Our differentiable room acoustic rendering framework [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Method Overview. Our framework contains two main components for rendering the room impulse response (RIR): (1) Visual Processing (top): Multi-view images of the scene are passed through a pre-trained vision encoder to extract pixel-aligned features at sampled points on the room surface. We then apply cross-attention both across views for each sampled point and across sampled points of query x to obtain a unified, ma… view at source ↗
Figure 3
Figure 3. Performance comparison across training scales (from [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Signal spatial distribution visualization. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Reflection response visualization. The RGB color encodes frequency-dependent reflection response, with red indicating high [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Acoustic beam tracing: in acoustic beam tracing the [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Local covariance derivation: as the traveling space [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visualization of surface basis samples for extracting [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Wave visualization on the Hearing Anything Anywhere dataset [ [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Wave visualization on the Real Acoustic Field dataset [ [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison across training scales (from [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: We visualize two failure cases in our model on RAF [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 68 canonical work pages

  1. [1]

    Yang, Brian Hamilton, Jonathan Sheaffer, Anurag Ranjan, Miguel Sarabia, Oncel Tuzel, and Jen-Hao Rick Chang

    Byeongjoo Ahn, Karren D. Yang, Brian Hamilton, Jonathan Sheaffer, Anurag Ranjan, Miguel Sarabia, Oncel Tuzel, and Jen-Hao Rick Chang. Novel-view acoustic synthesis from 3d reconstructed rooms. ArXiv, abs/2310.15130, 2023. 2

    arXiv 2023

  2. [2]

    Aerophones in flat- land: interactive wave simulation of wind instruments

    Andrew Allen and Nikunj Raghuvanshi. Aerophones in flat- land: interactive wave simulation of wind instruments. ACM Trans. Graph., 34(4), July 2015. 2

    work page 2015

  3. [3]

    Allen and David A

    Jont B. Allen and David A. Berkley. Image method for effi- ciently simulating small-room acoustics. The Journal of the Acoustical Society of America, 65(4):943–950, 04 1979. 2, 3

    work page 1979

  4. [4]

    Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P

    Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields. 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV) , pages 5835–5844, 2021. 5

    work page 2021

  5. [5]

    Av-gs: Learning material and ge- ometry aware priors for novel view acoustic synthesis

    Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Jiankang Deng, and Xiatian Zhu. Av-gs: Learning material and ge- ometry aware priors for novel view acoustic synthesis. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2

    work page 2024

  6. [6]

    Interactive sound propagation with bidirec- tional path tracing

    Chunxiao Cao, Zhong Ren, Carl Schissler, Dinesh Manocha, and Kun Zhou. Interactive sound propagation with bidirec- tional path tracing. ACM Trans. Graph., 35(6), Dec. 2016. 1

    work page 2016

  7. [7]

    Visual acoustic matching

    Changan Chen, Ruohan Gao, Paul T Calamia, and Kris- ten Grauman. Visual acoustic matching. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18836–18846, 2022. 2

    work page 2022

  8. [8]

    Learning to set waypoints for audio-visual navigation

    Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In In- ternational Conference on Learning Representations (ICLR),

    work page

Show all 73 references
  1. [9]

    Novel-view acoustic synthesis

    Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. In CVPR, 2023. 2

    2023

  2. [10]

    Soundspaces 2.0: A simulation platform for visual-acoustic learning

    Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. arXiv,

  3. [11]

    Learning audio-visual dereverberation

    Changan Chen, Wei Sun, David Harwath, and Kristen Grau- man. Learning audio-visual dereverberation. In ICASSP,

  4. [12]

    Be everywhere - hear everything (bee): Audio scene reconstruction by sparse audio-visual samples

    Mingfei Chen, Kun Su, and Eli Shlizerman. Be everywhere - hear everything (bee): Audio scene reconstruction by sparse audio-visual samples. 2023 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 7819–7828, 2023. 2

    2023

  5. [13]

    Real acoustic fields: An audio-visual room acous- tics dataset and benchmark

    Ziyang Chen, Israel D Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acous- tics dataset and benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2024

  6. [14]

    Structure from silence: Learning scene structure from ambient sound

    Ziyang Chen, Xixi Hu, and Andrew Owens. Structure from silence: Learning scene structure from ambient sound. Con- ference on Robot Learning (CoRL), 2021. 2

    2021

  7. [15]

    Sound localization from motion: Jointly learning sound direction and camera rotation

    Ziyang Chen, Shengyi Qian, and Andrew Owens. Sound localization from motion: Jointly learning sound direction and camera rotation. 2023. 2

    2023

  8. [16]

    Differentiable ocean acoustic propagation modeling

    Mandar Chitre. Differentiable ocean acoustic propagation modeling. In OCEANS 2023 - Limerick, pages 1–8, 2023. 3

    2023

  9. [17]

    Ad- verb: Visually guided audio dereverberation

    Sanjoy Chowdhury, Sreyan Ghosh, Dasgupta Subhrajyoti, Anton Ratnarajah, Utkarsh Tyagi, and Dinesh Manocha. Ad- verb: Visually guided audio dereverberation. ICCV, 2023. 2

    2023

  10. [18]

    Batvision: Learning to see 3d spatial layout with two ears

    Jesper Haahr Christensen, Sascha Hornauer, and X Yu Stella. Batvision: Learning to see 3d spatial layout with two ears. In ICRA. IEEE, 2020. 2

    2020

  11. [19]

    Diffimpact: Dif- ferentiable rendering and identification of impact sounds

    Samuel Clarke, Negin Heravi, Mark Rau, Ruohan Gao, Jia- jun Wu, Doug James, and Jeannette Bohg. Diffimpact: Dif- ferentiable rendering and identification of impact sounds. In 5th Annual Conference on Robot Learning, 2021. 3

    2021

  12. [20]

    Ddsp: Differentiable digital signal process- ing

    Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal process- ing. In International Conference on Learning Representa- tions, 2020. 3

    2020

  13. [21]

    A beam tracing approach to acoustic modeling for interactive virtual environments

    Thomas Funkhouser, Ingrid Carlbom, Gary Elko, Gopal Pin- gali, Mohan Sondhi, and Jim West. A beam tracing approach to acoustic modeling for interactive virtual environments. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ...

    1998

  14. [22]

    Look, listen, and act: Towards audio- visual embodied navigation

    Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio- visual embodied navigation. In ICRA, 2020. 2

    2020

  15. [23]

    Visualechoes: Spatial visual repre- sentation learning through echolocation

    Ruohan Gao, Changan Chen, Ziad Al-Halah, Carl Schissler, and Kristen Grauman. Visualechoes: Spatial visual repre- sentation learning through echolocation. In European Con- ference on Computer Vision (ECCV), 2020. 2

    2020

  16. [24]

    2.5d visual sound

    Ruohan Gao and Kristen Grauman. 2.5d visual sound. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  17. [25]

    Sonicverse: A multisensory simulation platform for training household agents that see and hear

    Ruohan Gao, Hao Li, Gokul Dharan, Zhuzhu Wang, Cheng- shu Li, Fei Xia, Silvio Savarese, Li Fei-Fei, and Jiajun Wu. Sonicverse: A multisensory simulation platform for training household agents that see and hear. In International Confer- ence on Robotics and Automation (ICRA),, 2023. 2

    2023

  18. [26]

    Geometry-aware multi-task learning for binaural audio gen- eration from video

    Rishabh Garg, Ruohan Gao, and Kristen Grauman. Geometry-aware multi-task learning for binaural audio gen- eration from video. In British Machine Vision Conference (BMVC), 2021. 2

    2021

  19. [27]

    Visually- guided audio spatialization in video with geometry-aware multi-task learning

    Rishabh Garg, Ruohan Gao, and Kristen Grauman. Visually- guided audio spatialization in video with geometry-aware multi-task learning. In International Journal of Computer Vision (IJCV), 2023. 2

    2023

  20. [28]

    Gumerov and Ramani Duraiswami

    Nail A. Gumerov and Ramani Duraiswami. A broadband fast multipole accelerated boundary element method for the three dimensional helmholtz equation. The Journal of the Acoustical Society of America, 125(1):191–205, 01 2009. 2

    2009

  21. [29]

    Hardin, Timothy Michaels, and Edward B

    Douglas P. Hardin, Timothy Michaels, and Edward B. Saff. A comparison of popular point configurations on S2. Dolomites Research Notes on Approximation, 9, 2016. 12

    2016

  22. [30]

    Thanedar

    John Kenneth Haviland and Balakrishna D. Thanedar. Monte carlo applications to acoustical field solutions. The Journal of the Acoustical Society of America , 54(6):1442–1448, 12

  23. [31]

    Zhang, Shaoqing Ren, and Jian Sun

    Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2015. 16

    2016

  24. [32]

    Diffsound: Differentiable modal sound rendering and inverse rendering for diverse inference tasks

    Xutong Jin, Chenxi Xu, Ruohan Gao, Jiajun Wu, Guoping Wang, and Sheng Li. Diffsound: Differentiable modal sound rendering and inverse rendering for diverse inference tasks. In SIGGRAPH, 2024. 3

    2024

  25. [33]

    Neu- ral 3d mesh renderer

    Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neu- ral 3d mesh renderer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3

    2018

  26. [34]

    Krokstad, S

    A. Krokstad, S. Strom, and S. Sørsdal. Calculating the acous- tical room response by the use of a ray tracing technique. Journal of Sound and Vibration, 8(1):118–125, 1968. 2, 3

    1968

  27. [35]

    Acoustic volume rendering for neural impulse re- sponse fields

    Zitong Lan, Chenhao Zheng, Zhiwei Zheng, and Mingmin Zhao. Acoustic volume rendering for neural impulse re- sponse fields. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 3, 6, 7, 15

    2024

  28. [36]

    Interactive sound rendering in complex and dynamic scenes using frustum tracing

    Christian Lauterbach, Anish Chandak, and Dinesh Manocha. Interactive sound rendering in complex and dynamic scenes using frustum tracing. IEEE Transactions on Visualization and Computer Graphics, 13:1672–1679, 2007. 2, 12

    2007

  29. [37]

    Scene-aware audio for 360 videos

    Dingzeyu Li, Timothy R Langlois, and Changxi Zheng. Scene-aware audio for 360 videos. ACM Transactions on Graphics (TOG), 37(4):1–12, 2018. 2

    2018

  30. [38]

    Self-supervised audio-visual soundscape stylization

    Tingle Li, Renhao Wang, Po-Yao Huang, Andrew Owens, and Gopala Anumanchipalli. Self-supervised audio-visual soundscape stylization. In ECCV, 2024. 2

    2024

  31. [39]

    Av-nerf: Learning neural fields for real-world audio-visual scene synthesis

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. In Conference on Neural Infor- mation Processing Systems (NeurIPS), 2023. 1, 2, 6, 7, 15

    2023

  32. [40]

    Neural acoustic context field: Rendering realistic room impulse response with neural fields

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Neural acoustic context field: Rendering realistic room impulse response with neural fields. ArXiv, abs/2309.15977, 2023. 1, 2, 13

    2023 arXiv

  33. [41]

    Amengual Gar´ı, Calvin Murdock, Ishwarya Ananthab- hotla, Philip Robinson, Eli Shlizerman, Vamsi Krishna Ithapu, and Ruohan Gao

    Xiulong Liu, Anurag Kumar, Paul Calamia, Sebasti `a V . Amengual Gar´ı, Calvin Murdock, Ishwarya Ananthab- hotla, Philip Robinson, Eli Shlizerman, Vamsi Krishna Ithapu, and Ruohan Gao. Hearing anywhere in any envi- ronment. In Conference on Computer Vision and Pattern Recognit...

    2025

  34. [42]

    Learning neural acoustic fields

    Andrew Luo, Yilun Du, Michael Tarr, Josh Tenenbaum, An- tonio Torralba, and Chuang Gan. Learning neural acoustic fields. Advances in Neural Information Processing Systems, 35:3165–3177, 2022. 1, 2, 6, 7, 15

    2022

  35. [43]

    Few-shot audio-visual learning of environ- ment acoustics

    Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kris- ten Grauman. Few-shot audio-visual learning of environ- ment acoustics. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 2, 13

    2022

  36. [44]

    Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duck- worth. NeRF in the Wild: Neural Radiance Fields for Un- constrained Photo Collections. In CVPR, 2021. 1

    2021

  37. [45]

    Gregory McDaniel and Cory L

    J. Gregory McDaniel and Cory L. Clarke. Interpretation and identification of minimum phase reflection coefficients. The Journal of the Acoustical Society of America , 110(6):3003– 3010, 12 2001. 4

    2001

  38. [46]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In ECCV, 2020. 1

    2020

  39. [47]

    Self-supervised generation of spatial audio for 360 video

    Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. Advances in neural information processing sys- tems, 2018. 2

    2018

  40. [48]

    Acoustic modeling using the digital waveg- uide mesh

    Damian Murphy, Antti Kelloniemi, Jack Mullen, and Si- mon Shelley. Acoustic modeling using the digital waveg- uide mesh. IEEE Signal Processing Magazine, 24(2):55–66,

  41. [49]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Ni...

    2023

  42. [50]

    Social connectedness in spatial audio calling contexts

    Vanessa Y Oviedo, Khia A Johnson, Madeline Huberth, and W Owen Brimijoin. Social connectedness in spatial audio calling contexts. Computers in Human Behavior Reports ,

  43. [51]

    Av-rir: Audio-visual room impulse response estimation

    Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar, Purva Chiniya, and Dinesh Manocha. Av-rir: Audio-visual room impulse response estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27164–27175, June 2024. 1, 2

    2024

  44. [52]

    Mesh2ir: Neural acoustic impulse response generator for complex 3d scenes

    Anton Ratnarajah, Zhenyu Tang, Rohith Aralikatti, and Di- nesh Manocha. Mesh2ir: Neural acoustic impulse response generator for complex 3d scenes. In Proceedings of the 30th ACM International Conference on Multimedia , MM ’22, page 924–933, New York, NY , USA, 2022. Associatio...

    2022

  45. [53]

    Fast-rir: Fast neural diffuse room impulse response generator

    Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir: Fast neural diffuse room impulse response generator. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 571–575, 2022. 2

    2022

  46. [54]

    Deep impulse responses: Estimating and parameterizing fil- ters with deep networks

    Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Estimating and parameterizing fil- ters with deep networks. ICASSP 2022 - 2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 3209–3213, 2022. 2

    2022

  47. [55]

    Peter Svensson

    Lauri Savioja and U. Peter Svensson. Overview of geomet- rical room acoustic modeling techniques. The Journal of the Acoustical Society of America, 138(2):708–730, 08 2015. 2, 3

    2015

  48. [56]

    High- order diffraction and diffuse reflections for interactive sound propagation in large environments

    Carl Schissler, Ravish Mehra, and Dinesh Manocha. High- order diffraction and diffuse reflections for interactive sound propagation in large environments. ACM Trans. Graph. , 33(4), July 2014. 2, 3

    2014

  49. [57]

    The room acoustic rendering equation

    Samuel Siltanen, Tapio Lokki, Sami Kiminki, and Lauri Savioja. The room acoustic rendering equation. The Journal of the Acoustical Society of America, 122:1624, 10 2007. 1

    2007

  50. [58]

    Image2reverb: Cross-modal reverb impulse response synthesis

    Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 286–295, October 2021. 2

    2021

  51. [59]

    Julius O. Smith. Physical modeling using digital waveguides. Computer Music Journal, 16(4):74–91, 1992. 2

    1992

  52. [60]

    Aria gen 2 case study: Envision - spatial audio navigation and ai research, 2025

    Meta Open Source. Aria gen 2 case study: Envision - spatial audio navigation and ai research, 2025. Accessed: 2025-04-

    2025

  53. [61]

    INRAS: Implicit neural representation for audio scenes

    Kun Su, Mingfei Chen, and Eli Shlizerman. INRAS: Implicit neural representation for audio scenes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 1, 2, 6, 7, 15

    2022

  54. [62]

    Thompson

    Lonny L. Thompson. A review of finite-element methods for time-harmonic acoustics. The Journal of the Acoustical Society of America, 119(3):1315–1330, 03 2006. 2

    2006

  55. [63]

    Efros, and Ji- tendra Malik

    Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Ji- tendra Malik. Multi-view supervision for single-view recon- struction via differentiable ray consistency. In Computer Vi- sion and Pattern Regognition (CVPR), 2017. 3

    2017

  56. [64]

    The prediction of echograms and impulse responses within the epidaure soft- ware

    Dirk van Maercke and Jacques Martin. The prediction of echograms and impulse responses within the epidaure soft- ware. Applied Acoustics, 38(2):93–114, 1993. 2, 3, 12

    1993

  57. [65]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, NIPS’17, page 6000–6010, R...

    2017

  58. [66]

    Soundcam: A dataset for tasks in tracking and identifying humans from real room acoustics

    Mason Wang, Samuel Clarke, Jui-Hsien Wang, Ruohan Gao, and Jiajun Wu. Soundcam: A dataset for tasks in tracking and identifying humans from real room acoustics. In Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2023. 2

    2023

  59. [67]

    Hearing anything any- where

    Mason Wang, Ryosuke Sawata, Samuel Clarke, Ruohan Gao, Shangzhe Wu, and Jiajun Wu. Hearing anything any- where. In CVPR, 2024. 1, 2, 3, 6, 7, 13, 14, 15

    2024

  60. [68]

    Zamir, Zhi-Yang He, Alexander Sax, Jiten- dra Malik, and Silvio Savarese

    Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jiten- dra Malik, and Silvio Savarese. Gibson env: real-world per- ception for embodied agents. In Computer Vision and Pat- tern Recognition (CVPR), 2018 IEEE Conference on. IEEE,

    2018

  61. [69]

    VR-NeRF: High-fidelity virtualized walkable spaces

    Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Bul `o, Lorenzo Porzi, Peter Kontschieder, Aljaˇz Boˇziˇc, Dahua Lin, Michael Zollh¨ofer, and Christian Richardt. VR-NeRF: High-fidelity virtualized walkable spaces. In SIGGRAPH Asia ...

    2023

  62. [70]

    Perspective transformer nets: Learning single- view 3d object reconstruction without 3d supervision

    Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single- view 3d object reconstruction without 3d supervision. In D. D. Lee, M. Sugiyama, U. V . Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information P...

  63. [71]

    Point transformer

    Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16259–16268, 2021. 5

    2021

  64. [72]

    Open3D: A modern library for 3D data processing

    Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847,

    arXiv

  65. [2018]

    stretches

    12 A.1. Method Details A.1.1. Acoustic Beam Tracing Algorithm Figure 6. Acoustic beam tracing: in acoustic beam tracing the source and listener are considered as two point, the sound is prop- agate via a cone-shape beam in space. Acoustic beam tracing han- dles reflection the ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.