{"id":"a1729f61-203a-40db-ad19-536795d1808b","arxiv_id":"1908.06752","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 360-degree audio-visual dataset and a deep-learning pipeline that estimates sound-source directions and encodes them into first-order Ambisonics for virtual reality.","lead":"This paper introduces the first dataset and pipeline for converting ordinary 360-degree videos with mono audio into full-sphere surround sound, called Ambisonics. It estimates where visible sound sources are located using audio and visual cues, then encodes those directions into the audio, which could make virtual reality content more immersive without special microphones.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4) requires N separated source signals s_i(t), but the pipeline input is mono/stereo and no separation step is described; multi-source Ambisonics generation is therefore undefined.","rationale":"The reader's weakest assumption (visibility-correspondence) is real and would corrupt the localization metrics if violated, but it is an external data-collection assumption that could be mitigated by re-annotation or dataset analysis. The missing source-separation step is internal to the method: even with perfect localization and perfect ground truth, Eq. (4) is not computable for N>1. The abstract and conclusion claim Ambisonics generation, while the experiments only evaluate localization. This strengthens the conditional verdict: the dataset and localization benchmark remain legitimate contributions, but the central 'generating Ambisonics' claim overreaches unless a separation stage is defined and the B-format output is evaluated. The reader's rationale already noted the absence of Ambisonics evaluation; this concern identifies the concrete mechanism behind that absence, namely that the individual source signals required by Eq. (4) are never produced. The conditional verdict remains appropriate because the omission is addressable in a revision rather than being a fundamental invalidation of the dataset or the localization study.","tokens_in":7671,"tokens_out":6237,"duration_ms":63249,"concrete_test":"Download the released code (github.com/V-Sense/360AudioVisual) and trace the data flow from the mono/stereo input to the B-format encoder. If there is no source-separation or per-source audio assignment module, construct a two-speaker test clip from the 360AVD annotations, run the pipeline, and compare the encoded WXYZ channels against a reference B-format (e.g., from an Ambisonic microphone or from oracle-separated speech placed at the annotated directions) using an objective metric such as the error in the active-intensity-vector direction over time. If s_i(t) is the same mono signal for all i, the output will point to one direction instead of two and the multi-source Ambisonics claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 defines first-order Ambisonics encoding as W = Σ s_i(t)/√2, X = Σ s_i(t) cos Φ_i cos θ_i, etc., where s_i(t) is 'the i-th sound signal of a given ODV' and N is the number of sound sources. Nowhere in the paper is a source-separation or signal-assignment stage described, and the input audio is a mono/stereo recording from a YouTube 360 video. For any clip with more than one active source (the dataset is described in Section 3 and the conclusion as containing multiple-speech scenarios), the s_i(t) terms cannot be obtained from the available audio. If the implementation silently sets every s_i(t) to the full mono mix, the resulting B-format collapses to one coincident source per time sample rather than a field with N sources. This is not a minor implementation detail: it is the difference between estimating directions and actually generating Ambisonics. The paper reports no evaluation of the generated B-format (no objective metric, no listening test, no comparison to a reference Ambisonic recording), so the central claim 'generating Ambisonics using audio-visual cue' is supported only up to the localization stage for a single-source case. The paper should either restrict the claim to single-source clips, describe and validate a separation stage, or present a B-format evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of generating first-order Ambisonics for 360-degree video from mono/stereo audio by visually localizing sound sources, together with a dataset of 265 360-degree videos with per-second 2D annotations of sound-source locations. It proposes a four-stage pipeline (spherical-to-planar representation, pre-trained audio-visual feature embedding, two prediction modules SsM and Att, and B-format encoding), and defines two evaluation metrics (360-SSD and 360-OvErr). Table 1 reports scores for the two representations and models, and the abstract claims that the cubical SsM variant is best. The paper closes with the caveat that the problem remains open.","tokens_in":7988,"tokens_out":5583,"duration_ms":46229,"significance":"The dataset release and the formulation of the task are useful first steps; the paper is, to the best of the authors' knowledge, the first to pose audio-visual Ambisonics estimation for ODVs. The authors also release code and use public pre-trained models, which supports reproducibility. The concrete comparison of two input representations is also a strength. However, the central claim of 'generating Ambisonics' is not substantiated by the reported experiments: no baseline contextualizes the localization numbers, the construction of 3D ground truth is not described, and Eq. (4) requires unavailable separated source signals. The work is therefore best read as an initial localization benchmark rather than a validated Ambisonics-generation result.","major_comments":[{"comment":"Equation (4) sums N distinct source signals s_i(t) with the predicted angles, but the input to the pipeline is the mono/stereo track of a YouTube clip and no source-separation or signal-assignment stage is described anywhere in Sections 4 or 5. As the conclusion notes that the dataset contains single-to-multiple speech scenarios, the s_i(t) are not defined for multi-source clips; if each s_i(t) is silently set to the full mix, the output collapses to a one-direction field rather than N spatially distributed sources. Please either restrict the claim to single-source clips, describe and validate a separation stage, or present a B-format evaluation that is sensitive to the multi-source structure.","section":"4.4, Eq. (4)"},{"comment":"All conclusions about the superiority of SsM-Cubical are based on absolute values without any baseline such as random guessing, a constant center-of-sphere predictor, an audio-only DOA estimator, or a saliency-based visual predictor. Without such a reference, 360-SSD between 0.71 and 0.79 cannot be interpreted as good or poor, and the per-model standard deviations overlap across rows. Add at least one simple baseline and report pairwise comparison or error bars on the comparison.","section":"5.3, Table 1"},{"comment":"The dataset section describes 2D pixel labels obtained with Microsoft VoTT, but the metrics and learning signal use 3D probability volumes ('spherical volumes S') and 'probability spheres of radius 0.5'. The paper does not specify how a 2D annotation in an equirectangular or cubemap frame is converted to a 3D direction, how the volume around that direction is constructed, how many sources N are assumed per second, or how the threshold epsilon maps to physical radii. This missing description makes the ground-truth volumes and hence both metrics undefined for a reader.","section":"3 and 4.3"},{"comment":"No direct evaluation of the generated B-format is reported: no objective comparison against a reference Ambisonic recording, no metric on the W/X/Y/Z channels, and no listening test. Even if the localization stage were validated, the abstract-level claim of 'generating Ambisonics' is only supported up to localization. The authors should either provide such an evaluation or explicitly position the contribution as sound-source localization for future B-format encoding.","section":"5.2 and 5.3"},{"comment":"The reported prediction modules are adopted from models pre-trained on ImageNet, AudioSet, 750k videos, and AVE, and no fine-tuning or train/validation/test split on 360AVD is described. Since the dataset is released 'to benchmark' the task, please specify whether the numbers are zero-shot transfer scores, define the split, and state whether any parameter is fit on the 265 clips; otherwise the dataset does not yet serve as a benchmark in the usual sense.","section":"5.2"}],"minor_comments":[{"comment":"The sentence 'The first projection, however, contains less geometrical distortions than the second one' is the opposite of the usual property and contradicts the discussion in Section 5.3 that cubical representation has 'less distortions'. Please correct the wording.","section":"4.1"},{"comment":"The mapping f is written as (x,y)->(theta,phi), while Eq. (4) uses azimuth/elevation notation Phi and theta in a different ordering; align the notation and define the coordinate convention once and consistently.","section":"Eqs. (1)-(4)"},{"comment":"Equation (3) is a notational abbreviation rather than an equation; please write the definition of (Phi_i, theta_i) from the center of the thresholded volume explicitly.","section":"Eq. (3)"},{"comment":"Indicate the number of clips used per cell and whether the average over 265 ODVs includes clips with multiple annotated sources; also state whether the metrics are computed per source or per clip.","section":"Table 1"},{"comment":"The reference to the released dataset and code is a GitHub URL without version or persistent identifier; please add a DOI or versioned landing page for archival reproducibility.","section":"Dataset availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is suitable in scope for a signal-processing venue but currently overclaims: the abstract promises Ambisonics generation while the evaluation stops at localization, and Eq. (4) is under-specified for multi-source audio. The dataset and idea are worth pursuing, so I recommend revision rather than rejection. Please also ask the authors to clarify, if possible, whether the Facebook 360 encoder received one point per source or a single mix; this determines whether the implementation matches Eq. (4)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the 360AVD dataset (265 360-degree clips with per-second sound-source annotations) is genuinely new, and the problem — generating Ambisonics from mono video audio using visual cues — is a real, currently unmet need in VR content creation. Second, the central empirical claim is not supported as written: the Ambisonics encoding equation requires N separated source signals, no separation stage is described or validated, and no baseline or B-format evaluation appears anywhere.\n\nWhat is actually good here: the dataset is the first of its kind, covering presentations, discussions, debates, and documentaries with per-second annotations. The paper honestly frames the work as an initial study, and the conclusion openly says the problem remains open. The equirectangular-versus-cubemap comparison is a reasonable sanity check, and the proposed metrics (360-SSD, 360-OvErr) are acceptable as relative measures.\n\nThe soft spots, in proportion: no baseline means the Table 1 scores (360-SSD 0.71 for SsM-Cubical) are uninterpretable — a random-origin or center-of-sphere baseline would give them meaning. The ground-truth construction is opaque: VoTT pixel annotations become 3D probability volumes, but the conversion is never described. The load-bearing issue is Eq. (4): it sums N distinct source signals s_i(t), yet the input is mono/stereo and no source-separation module exists. For multi-speaker clips, which the dataset explicitly contains, those s_i(t) are undefined. If the implementation simply uses the full mix for every source, the B-format collapses to one coincident source per time sample, and the claimed \"Ambisonics generation\" reduces to direction estimation with an extra encoder. No listening test, no objective B-format metric, and no comparison against a reference Ambisonic recording are reported. The visibility assumption — every sound source appears in the 360 frame — is also unexamined; off-screen sounds and unannotated sources would corrupt the ground truth.\n\nThese flaws are addressable. The dataset and problem formulation are legitimate, and a revised paper with baselines, a described separation stage (or a restricted single-source claim), and a real B-format evaluation would be worth serious attention. As submitted, the paper overstates what it demonstrates, but it is an honest and useful opening statement of a new research direction.\n\nRecommendation: send it to peer review. The dataset and problem will interest the multimedia and VR communities, and referees can demand the missing experiments. For my own work, I would not cite the current results, but I would watch for a revised version.","headline":"New dataset and problem formulation, but the Ambisonics encoding step is underspecified and the empirical claims outrun the evidence.","tokens_in":8479,"tokens_out":2566,"would_cite":false,"duration_ms":26349,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Localizing visible speakers turns mono 360 video into Ambisonics","keywords":["Ambisonics","360-degree video","audio-visual learning","sound source localization","virtual reality","B-format","omnidirectional video","benchmark dataset"],"falsifier":"Record a 360-degree scene with a talker outside the camera's field of view while an on-screen person stays silent, run the pipeline, and compare the predicted B-format direction with the true sound direction; if the pipeline points at the visible person, the central claim is falsified for the common off-screen-source case.","tokens_in":7509,"feed_emoji":"🎧","tokens_out":6724,"duration_ms":64313,"temperature":0.7,"pith_summary":"The paper introduces a new task: turning the mono or stereo audio track of a 360-degree video into first-order Ambisonics by locating sound sources through audio-visual cues. It argues that deep audio-visual feature embeddings can predict where visible speakers are on the viewing sphere, and that those predicted locations can drive B-format encoding, producing full-sphere spatial sound without a sound-field microphone. To support this, it contributes a dataset of 265 annotated 10-second clips and a four-stage pipeline spanning projection, feature embedding, prediction, and encoding. The best configuration, a self-supervised prediction module on cubic projection, localizes sound-source centers at 0.71 normalized distance error, and the paper claims this validates the direction while leaving room for improvement.","feed_headline":"Localizing visible speakers turns mono 360 video into Ambisonics","feed_subtitle":"Ordinary 360 cameras could gain spatial audio by letting the image locate sound sources, no sound-field mic needed.","key_machinery":"The load-bearing mechanism is the prediction-to-encoding chain: concatenated audio-visual features are mapped through a learned fusion network into a probability volume over the sphere; the volume's center becomes the sound-source direction, which is then plugged into the first-order Ambisonics B-format equations (W is the omnidirectional component, and X, Y, Z are the Cartesian direction components weighted by the estimated azimuth and elevation). A second key choice is the sphere-to-plane projection used as input: cubemap projection, built from six faces, produces less distortion and consistently better localization than equirectangular projection in the reported experiments.","core_discovery":"The central claim is that 3D sound-source location for an omnidirectional video can be estimated automatically from the correlation between the mono audio and the video frames, and that this estimate is accurate enough to encode usable first-order Ambisonics. The pipeline first represents the sphere as either an equirectangular or cubemap image, extracts separate audio and visual feature embeddings, concatenates them, and lets one of two prediction modules (a self-supervised convolutional fusion or an attention-based fusion) produce a spherical probability volume for the sound source. The center of the thresholded volume is converted to spherical coordinates and substituted into the B-format equations; the four resulting channels (W, X, Y, Z) are the Ambisonics output. The paper reports that the self-supervised module with cubemap input gives the lowest localization distance and overlap errors on the 265-video dataset, and that cubemap consistently beats equirectangular because of lower geometric distortion.","pith_inferences":["Editorial inference: the ground-truth annotation marks only visible sources, so the reported accuracy probably degrades when the audio contains off-screen sounds, background music, or unannotated overlapping voices; the paper does not test these cases.","Editorial inference: the prediction modules are transferred from 2D video models without fine-tuning on 360 projections, so the reported errors are likely an upper bound for what the architecture can achieve with end-to-end training.","Editorial inference: the B-format encoder represents each source as one point, so simultaneous speakers arriving from nearly the same direction would be merged; the method is best suited to scenes with a few spatially separated talkers.","Editorial inference: the paper evaluates localization rather than the perceived quality of the rendered spatial audio, so an even stronger test would compare the generated B-format against a concurrent recording from a real Ambisonic microphone."],"forward_implications":["A 360-degree camera that records only mono audio could automatically produce first-order Ambisonics for VR playback, removing the sound-field microphone from the capture chain.","Cubemap projection is the stronger input representation for this task, so future 360 audio-visual localization systems should build on six-face projections rather than equirectangular frames.","The 265-video annotated dataset and the two proposed metrics give the field a common benchmark for automatic spatial-audio generation.","Because the pipeline outputs a per-second 3D probability volume, the same prediction could also steer directional audio effects or guide attention in omnidirectional video."],"supporting_citations":[{"why":"Defines the B-format signal equations that the encoding stage uses for the W, X, Y, and Z channels.","marker":"[11]"},{"why":"Supplies the self-supervised fusion sub-network and its pretrained layers used by the SsM prediction module.","marker":"[12]"},{"why":"Supplies the attention-based fusion mechanism and transformation layers used by the Att prediction module.","marker":"[13]"},{"why":"Provides the deep convolutional visual feature extractor applied to the projected 360 frames.","marker":"[19]"},{"why":"Provides the deep audio feature extractor that produces a 128-dimensional embedding for each second of sound.","marker":"[20]"},{"why":"Defines the single-object localization error that the proposed 360-Overlap-Error adapts to 3D probability volumes.","marker":"[21]"},{"why":"Provides the spatial-encoding tool that places the predicted source locations into the B-format channels.","marker":"[23]"}],"fun_headline_variants":["AI locates sound sources in 360 video using visual cues for Ambisonics","Visual cues let mono 360 video become full-sphere Ambisonics audio","No mic needed: AI uses video to place sound in 360 video as Ambisonics","Visual input helps AI place audio sources in 360 video for spatial sound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every sound source that matters is visible in the 360-degree frame and has been manually marked, so the ground-truth locations reflect what the audio truly contains.","fun_headline_variants_meta":{"raw":{"variants":["AI locates sound sources in 360 video using visual cues for Ambisonics","Visual cues let mono 360 video become full-sphere Ambisonics audio","No mic needed: AI uses video to place sound in 360 video as Ambisonics","Visual input helps AI place audio sources in 360 video for spatial sound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3122,"prompt_tokens":944,"completion_tokens":2178,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2092}},"tokens_in":560,"tokens_out":2178,"duration_ms":15819,"temperature":1.0,"reasoning_tokens":2092,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:58:03.572314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a 360-degree scene with a talker outside the camera's field of view while an on-screen person stays silent, run the pipeline, and compare the predicted B-format direction with the true sound direction; if the pipeline points at the visible person, the central claim is falsified for the common off-screen-source case.","supporting_citations":[{"cited_title":"Ambisonics in multichannel broadcasting and video,","cited_arxiv_id":null,"evidence_quote":"Defines the B-format signal equations that the encoding stage uses for the W, X, Y, and Z channels."},{"cited_title":"Audio-visual scene analysis with self- supervised multisensory features,","cited_arxiv_id":null,"evidence_quote":"Supplies the self-supervised fusion sub-network and its pretrained layers used by the SsM prediction module."},{"cited_title":"Audio- visual event localization in unconstrained videos,","cited_arxiv_id":null,"evidence_quote":"Supplies the attention-based fusion mechanism and transformation layers used by the Att prediction module."},{"cited_title":"Very deep convolutional networks for large-scale image recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the deep convolutional visual feature extractor applied to the projected 360 frames."},{"cited_title":"Imagenet large scale visual recognition challenge,","cited_arxiv_id":null,"evidence_quote":"Defines the single-object localization error that the proposed 360-Overlap-Error adapts to 3D probability volumes."},{"cited_title":"The Facebook 360 Spatial Workstation,","cited_arxiv_id":null,"evidence_quote":"Provides the spatial-encoding tool that places the predicted source locations into the B-format channels."}],"review_version":1}