REVIEW 5 major objections 5 minor 24 references
Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality
T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Localizing visible speakers turns mono 360 video into Ambisonics
desk verdict New dataset and problem formulation, but the Ambisonics encoding step is underspecified and the empirical claims outrun the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the prediction-to-encoding chain: concatenated audio-visual features are mapped through a learned fusion network into a probability volume over the sphere; the volume's center becomes the sound-source direction, which is then plugged into the first-order Ambisonics B-format equations (W is the omnidirectional component, and X, Y, Z are the Cartesian direction components weighted by the estimated azimuth and elevation). A second key choice is the sphere-to-plane projection used as input: cubemap projection, built from six faces, produces less distortion and consistently better localization than equirectangular projection in the reported experiments.
What would settle it
Record a 360-degree scene with a talker outside the camera's field of view while an on-screen person stays silent, run the pipeline, and compare the predicted B-format direction with the true sound direction; if the pipeline points at the visible person, the central claim is falsified for the common off-screen-source case.
Extended reading notes
Core claim
The central claim is that 3D sound-source location for an omnidirectional video can be estimated automatically from the correlation between the mono audio and the video frames, and that this estimate is accurate enough to encode usable first-order Ambisonics. The pipeline first represents the sphere as either an equirectangular or cubemap image, extracts separate audio and visual feature embeddings, concatenates them, and lets one of two prediction modules (a self-supervised convolutional fusion or an attention-based fusion) produce a spherical probability volume for the sound source. The center of the thresholded volume is converted to spherical coordinates and substituted into the B-format equations; the four resulting channels (W, X, Y, Z) are the Ambisonics output. The paper reports that the self-supervised module with cubemap input gives the lowest localization distance and overlap errors on the 265-video dataset, and that cubemap consistently beats equirectangular because of lower geometric distortion.
Load-bearing premise
Every sound source that matters is visible in the 360-degree frame and has been manually marked, so the ground-truth locations reflect what the audio truly contains.
Editorial extensions
If this is right
- A 360-degree camera that records only mono audio could automatically produce first-order Ambisonics for VR playback, removing the sound-field microphone from the capture chain.
- Cubemap projection is the stronger input representation for this task, so future 360 audio-visual localization systems should build on six-face projections rather than equirectangular frames.
- The 265-video annotated dataset and the two proposed metrics give the field a common benchmark for automatic spatial-audio generation.
- Because the pipeline outputs a per-second 3D probability volume, the same prediction could also steer directional audio effects or guide attention in omnidirectional video.
Reading between the lines
- Editorial inference: the ground-truth annotation marks only visible sources, so the reported accuracy probably degrades when the audio contains off-screen sounds, background music, or unannotated overlapping voices; the paper does not test these cases.
- Editorial inference: the prediction modules are transferred from 2D video models without fine-tuning on 360 projections, so the reported errors are likely an upper bound for what the architecture can achieve with end-to-end training.
- Editorial inference: the B-format encoder represents each source as one point, so simultaneous speakers arriving from nearly the same direction would be merged; the method is best suited to scenes with a few spatially separated talkers.
- Editorial inference: the paper evaluates localization rather than the perceived quality of the rendered spatial audio, so an even stronger test would compare the generated B-format against a concurrent recording from a real Ambisonic microphone.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the task of generating first-order Ambisonics for 360-degree video from mono/stereo audio by visually localizing sound sources, together with a dataset of 265 360-degree videos with per-second 2D annotations of sound-source locations. It proposes a four-stage pipeline (spherical-to-planar representation, pre-trained audio-visual feature embedding, two prediction modules SsM and Att, and B-format encoding), and defines two evaluation metrics (360-SSD and 360-OvErr). Table 1 reports scores for the two representations and models, and the abstract claims that the cubical SsM variant is best. The paper closes with the caveat that the problem remains open.
Significance. The dataset release and the formulation of the task are useful first steps; the paper is, to the best of the authors' knowledge, the first to pose audio-visual Ambisonics estimation for ODVs. The authors also release code and use public pre-trained models, which supports reproducibility. The concrete comparison of two input representations is also a strength. However, the central claim of 'generating Ambisonics' is not substantiated by the reported experiments: no baseline contextualizes the localization numbers, the construction of 3D ground truth is not described, and Eq. (4) requires unavailable separated source signals. The work is therefore best read as an initial localization benchmark rather than a validated Ambisonics-generation result.
major comments (5)
- [4.4, Eq. (4)] Equation (4) sums N distinct source signals s_i(t) with the predicted angles, but the input to the pipeline is the mono/stereo track of a YouTube clip and no source-separation or signal-assignment stage is described anywhere in Sections 4 or 5. As the conclusion notes that the dataset contains single-to-multiple speech scenarios, the s_i(t) are not defined for multi-source clips; if each s_i(t) is silently set to the full mix, the output collapses to a one-direction field rather than N spatially distributed sources. Please either restrict the claim to single-source clips, describe and validate a separation stage, or present a B-format evaluation that is sensitive to the multi-source structure.
- [5.3, Table 1] All conclusions about the superiority of SsM-Cubical are based on absolute values without any baseline such as random guessing, a constant center-of-sphere predictor, an audio-only DOA estimator, or a saliency-based visual predictor. Without such a reference, 360-SSD between 0.71 and 0.79 cannot be interpreted as good or poor, and the per-model standard deviations overlap across rows. Add at least one simple baseline and report pairwise comparison or error bars on the comparison.
- [3 and 4.3] The dataset section describes 2D pixel labels obtained with Microsoft VoTT, but the metrics and learning signal use 3D probability volumes ('spherical volumes S') and 'probability spheres of radius 0.5'. The paper does not specify how a 2D annotation in an equirectangular or cubemap frame is converted to a 3D direction, how the volume around that direction is constructed, how many sources N are assumed per second, or how the threshold epsilon maps to physical radii. This missing description makes the ground-truth volumes and hence both metrics undefined for a reader.
- [5.2 and 5.3] No direct evaluation of the generated B-format is reported: no objective comparison against a reference Ambisonic recording, no metric on the W/X/Y/Z channels, and no listening test. Even if the localization stage were validated, the abstract-level claim of 'generating Ambisonics' is only supported up to localization. The authors should either provide such an evaluation or explicitly position the contribution as sound-source localization for future B-format encoding.
- [5.2] The reported prediction modules are adopted from models pre-trained on ImageNet, AudioSet, 750k videos, and AVE, and no fine-tuning or train/validation/test split on 360AVD is described. Since the dataset is released 'to benchmark' the task, please specify whether the numbers are zero-shot transfer scores, define the split, and state whether any parameter is fit on the 265 clips; otherwise the dataset does not yet serve as a benchmark in the usual sense.
minor comments (5)
- [4.1] The sentence 'The first projection, however, contains less geometrical distortions than the second one' is the opposite of the usual property and contradicts the discussion in Section 5.3 that cubical representation has 'less distortions'. Please correct the wording.
- [Eqs. (1)-(4)] The mapping f is written as (x,y)->(theta,phi), while Eq. (4) uses azimuth/elevation notation Phi and theta in a different ordering; align the notation and define the coordinate convention once and consistently.
- [Eq. (3)] Equation (3) is a notational abbreviation rather than an equation; please write the definition of (Phi_i, theta_i) from the center of the thresholded volume explicitly.
- [Table 1] Indicate the number of clips used per cell and whether the average over 265 ODVs includes clips with multiple annotated sources; also state whether the metrics are computed per source or per clip.
- [Dataset availability] The reference to the released dataset and code is a GitHub URL without version or persistent identifier; please add a DOI or versioned landing page for archival reproducibility.
Circularity Check
No significant circularity: predictions are benchmarked with pre-trained external models and the reported metrics are not forced by construction.
full rationale
The paper's central claim is an empirical localization benchmark on the new 360AVD dataset, and the prediction modules are adopted from externally trained models (Owens-Efros trained on 750k videos; Tian et al. trained on AVE; VGG-19 on ImageNet; VGG-like audio network on AudioSet), with no fitting of those models to the target dataset. The reported 360-SSD and 360-OvErr scores could have come out badly, so the comparison is not predetermined. The B-format equations in Eq. (4) are the standard Ambisonics encoding given estimated directions and source signals; they do not define the inputs in terms of the output. The absence of a described source-separation stage for multi-source clips is a real correctness/completeness gap (the paper even concludes the problem 'remains open'), but it is not a circularity: nothing in the derivation is equivalent to its own input by construction. Self-citations to the authors' earlier VR attention papers are contextual and not load-bearing for the localization or Ambisonics claims. Therefore the derivation chain is self-contained with respect to the external benchmarks, and no circular step can be exhibited from the text.
Assumptions & free parameters
free parameters (3)
- epsilon (epsilon) threshold =
0.4, 0.5, 0.6 (swept)
- probability sphere radius =
0.5
- number of sound sources N =
not estimated; fixed by dataset annotation
assumptions (4)
- domain assumption Every sound source in the audio is visible in the 360-degree frame and the manual 2D annotations correspond to the true 3D sound-source positions.
- domain assumption Pre-trained 2D audio-visual models (Owens et al., Tian et al.) transfer to 360-degree equirectangular and cubemap projections without fine-tuning.
- standard math Ambisonics B-format encoding equations (Gerzon) are the correct representation for the spatial audio task.
- domain assumption The audio track in each YouTube 360 video corresponds to the visible scene and contains no significant non-visual sources.
Cite this review
Pith. "Pith review of Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality." pith.science (2026). https://pith.science/paper/OCLOT3FN
@misc{pith2026190806752,
author = {Pith},
title = {Pith review of: Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality},
year = {2026},
howpublished = {\url{https://pith.science/paper/OCLOT3FN}},
note = {Machine review of arXiv:1908.06752}
}
read the original abstract
Ambisonics i.e., a full-sphere surround sound, is quintessential with 360-degree visual content to provide a realistic virtual reality (VR) experience. While 360-degree visual content capture gained a tremendous boost recently, the estimation of corresponding spatial sound is still challenging due to the required sound-field microphones or information about the sound-source locations. In this paper, we introduce a novel problem of generating Ambisonics in 360-degree videos using the audio-visual cue. With this aim, firstly, a novel 360-degree audio-visual video dataset of 265 videos is introduced with annotated sound-source locations. Secondly, a pipeline is designed for an automatic Ambisonic estimation problem. Benefiting from the deep learning-based audio-visual feature-embedding and prediction modules, our pipeline estimates the 3D sound-source locations and further use such locations to encode to the B-format. To benchmark our dataset and pipeline, we additionally propose evaluation criteria to investigate the performance using different 360-degree input representations. Our results demonstrate the efficacy of the proposed pipeline and open up a new area of research in 360-degree audio-visual analysis for future investigations.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
“Samsung Gear 360,” https://www.samsung.com/global/ galaxy/gear-360/, Accessed: 2018-10-27
work page 2018
-
[3]
Director’s cut - analysis of aspects of interactive storytelling for vr films,
C. O Fearghail, C. Ozcinar, S. Knorr, and A. Smolic, “Director’s cut - analysis of aspects of interactive storytelling for vr films,” in Inter- national Conference for Interactive Digital Storytelling (ICIDS) 2018 , 2018
work page 2018
-
[4]
Director’s cut - a combined dataset for visual attention analysis in cinematic vr content,
S. Knorr, C. Ozcinar, C. O Fearghail, and A. Smolic, “Director’s cut - a combined dataset for visual attention analysis in cinematic vr content,” in The 15th ACM SIGGRAPH European Conference on Visual Media Production (CVMP), 2018
work page 2018
-
[5]
Visual attention in omnidirectional video for virtual reality applications,
C. Ozcinar and A. Smolic, “Visual attention in omnidirectional video for virtual reality applications,” in 10th International Conference on Quality of Multimedia Experience (QoMEX), 2018
work page 2018
-
[6]
Beckett in vr: Exploring narrative using free viewpoint video,
N. O’Dwyer, N. Johnson, R. Pag´es, J. Ondˇrej, K. Amplianitis, E. Bates, D. Monaghan, and A. Smoli´c, “Beckett in vr: Exploring narrative using free viewpoint video,” in ACM SIGGRAPH 2018 Posters, New York, NY , USA, 2018, SIGGRAPH ’18, pp. 2:1–2:2, ACM
work page 2018
-
[7]
Saliency in vr: How do people explore virtual environments?,
V . Sitzmann, A. Serrano, A. Pavel, M. Agrawala, D. Gutierrez, B. Ma- sia, and G. Wetzstein, “Saliency in vr: How do people explore virtual environments?,” IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 4, pp. 1633–1642, April 2018
work page 2018
-
[8]
Producing 3d audio in ambison- ics,
M. Frank, F. Zotter, and A. Sontacchi, “Producing 3d audio in ambison- ics,” in Audio Engineering Society Conference: 57th International Conference: The Future of Audio Entertainment Technology–Cinema, Television and the Internet. Audio Engineering Society, 2015
work page 2015
Show all 24 references
-
[9]
Sound localisation for 3d multimedia streaming,
H. Lim, C. Ozcinar, A. P. Hill, and A. Kondoz, “Sound localisation for 3d multimedia streaming,” in Proceedings of the 12th Western Pacific Acoustics Conference (WESPAC), 2015
2015
-
[10]
Comparing ambisonic microphones [blog],
E. Bates, “Comparing ambisonic microphones [blog],” https://endabates.wordpress.com/2017/06/19/ comparing-ambisonic-microphones/, Accessed: 2018-10- 27
2017
-
[11]
Ambisonics in multichannel broadcasting and video,
M. A. Gerzon, “Ambisonics in multichannel broadcasting and video,” Journal of the Audio Engineering Society. Audio Engineering Society , vol. 33, no. 11, pp. 859–871, 1985
1985
-
[12]
Audio-visual scene analysis with self- supervised multisensory features,
A. Owens and A. A. Efros, “Audio-visual scene analysis with self- supervised multisensory features,” European Conference on Computer Vision (ECCV), 2018
2018
-
[13]
Audio- visual event localization in unconstrained videos,
T. Yapeng, S. Jing, Bochen L., D. Zhiyao, and X. Chenliang, “Audio- visual event localization in unconstrained videos,” in The European Conference on Computer Vision (ECCV), September 2018
2018
-
[14]
Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation,
A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation,” ACM Transactions on Graphics, July 2018
2018
-
[15]
Predicting audio-visual salient events based on vi- sual, audio and text modalities for movie summarization,
P. Koutras, A. Zlatintsi, E. Iosif, A. Katsamanis, P. Maragos, and A. Potamianos, “Predicting audio-visual salient events based on vi- sual, audio and text modalities for movie summarization,” in IEEE International Conference on Image Processing (ICIP) , Sept 2015, pp. 4361–4365
2015
-
[16]
An Audio-Visual system for Object-Based audio: From recording to listen- ing,
P. Coleman, A. Franck, J Francombe, Q Liu, T. de Campos, R. J. Hughes, D. Menzies, M F S G ´alvez, Y . Tang, J. Woodcock, P. J. B. Jackson, F. Melchior, C. Pike, F. M. Fazi, T. J. Cox, and A. Hilton, “An Audio-Visual system for Object-Based audio: From recording to listen- ing...
1919
-
[17]
Multichannel speech separation with recurrent neural networks from high-order ambisonics recordings,
L. Perotin, R. Serizel, E. Vincent, and A. Gurin, “Multichannel speech separation with recurrent neural networks from high-order ambisonics recordings,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 36–40
2018
-
[18]
Feature Learning for the Image Retrieval Task,
A. Rana, J. Zepeda, and P. P ´erez, “Feature Learning for the Image Retrieval Task,” inComputer Vision - FSLCV , ACCV 2014 - Singapore, November 1-2, 2014, 2014, pp. 152–165
2014
-
[19]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” inInternational Conference on Ma- chine Learning, 2015
2015
-
[20]
CNN architectures for large-scale audio classification,
S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. Channing Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” CoRR, vol. abs/1609.09430, 2016
2016 arXiv
-
[21]
Imagenet large scale visual recognition challenge,
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” In- ternational Journal of Computer Vision, vol. 115, no. 3, Dec. 2015
2015
-
[22]
Audio set: An ontology and human-labeled dataset for audio events,
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in IEEE International Conference on Acoustics, Speech and Signal Pro- cessi...
2017
-
[23]
The Facebook 360 Spatial Workstation,
“The Facebook 360 Spatial Workstation,” https:// facebook360.fb.com/spatial-workstation/, Accessed: 2018-10-27
2018
-
[24]
Experimenting with multimedia advances using GPAC,
J. Le Feuvre, C. Concolato, J.-C. Dufourd, R. Bouqueau, and J.-C. Moissinac, “Experimenting with multimedia advances using GPAC,” in Proceedings of the 19th ACM International Conference on Multime- dia, New York, NY , USA, 2011, MM ’11, pp. 715–718, ACM
2011
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.