REVIEW 3 major objections 5 minor 34 references
SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read SoundSculpt extracts a target sound from ambisonic recordings by combining the sound's direction with a semantic description, and it beats signal-processing baselines, especially when an interfering source is close by.
desk verdict A useful first ambisonic-in-ambisonic target extender with a clean architecture, but the real-data evidence for the semantic benefit is built on an artificial test construction that deserves scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a first-order ambisonic-to-ambisonic U-Net with four input and output channels (W, X, Y, Z), extended from a mono-to-mono semantically conditioned U-Net. Conditioning is applied through feature-wise linear modulation (FiLM): the target direction is projected to a 128-dimensional vector, the semantic embedding (from SoundWords or BERT) is separately projected to 128 dimensions, and the two are added before modulating the encoder and decoder layers. The training loss is the L1 distance between the complex short-time Fourier transforms of the target and the estimate, averaged over channels. The synthetic training data relies on image-source method room impulse responses convolved with source waveforms, while real data comes from rotated segments of the STARSS23 dataset.
What would settle it
A concrete test would be to record real acoustic scenes with a spherical microphone array where the target and a secondary source are truly within 15 degrees of each other and their directions are precisely known, then compare spatial-only versus spatial-plus-semantic extraction. If the combined model does not show a significant SI-SDRi improvement over spatial-only in those real close-secondary conditions, the paper's central claim would be weakened.
Extended reading notes
Core claim
The paper's central claim is that a network conditioned on both the target direction (azimuth and elevation) and a semantic embedding of the target sound can extract the target ambisonic signal from a mixture with markedly higher SI-SDRi than three signal-processing baselines: ambisonic loudness modification, max-DI beamform-and-project, and max-rE beamform-and-project. On synthetic test mixtures, the spatial-plus-semantic model reaches 10.73 dB SI-SDRi (with the SoundWords encoder) versus 9.36 dB for spatial-only and 2.47 dB on real STARSS23 mixtures versus -0.59 dB with direction alone. The paper also shows that the semantic cue adds the most value when a secondary source is within 15 degrees of the target: on the STARSS23 test set, spatial-plus-SoundWords improves over spatial-only by about 1.8 dB in that condition, while being slightly worse when no close secondary source is present. Finally, the paper compares two text encoders, BERT and the audio-text contrastive SoundWords encoder, and finds that the audio-aligned SoundWords gives better results on real mixtures.
Load-bearing premise
The evaluation assumes that the synthetic image-source room impulse responses and the rotation-based manipulation of real STARSS23 segments faithfully represent real ambisonic recordings, including the 50% probability of a nearby secondary source in test mixtures.
Editorial extensions
If this is right
- If the claim holds, users of ambisonic recordings (VR/AR, immersive video) could isolate a target sound by pointing and describing, preserving its spatial position and reverberation, in real time.
- The result suggests that semantic conditioning is not merely a substitute for spatial information but a complementary cue that resolves ambiguity when sources are angularly close.
- The comparison of text encoders indicates that an audio-text contrastive embedding (SoundWords) transfers better to real recordings than a text-only embedding (BERT), pointing to future work on audio-aligned semantic representations.
- The strong synthetic-to-real gap (10.73 dB vs 2.47 dB) suggests that domain mismatch in ambisonic mixture simulation is a key limitation that future methods must address.
Reading between the lines
- The paper's core idea could extend beyond first-order ambisonics to higher-order ambisonics or binaural rendering, where preserving spatial cues matters for head-tracked playback.
- The close-secondary result hints that semantic conditioning may also help in other source-separation tasks with spatially overlapping or adjacent sources, such as speech separation with co-located talkers.
- A testable extension is to evaluate the method on real recordings with precise, independently measured source directions and no rotation augmentation; the claimed benefit of semantic conditioning in close-secondary scenes should be verified under those conditions.
- The 50% probability of placing a nearby secondary source in the test mixtures may inflate the apparent utility of semantic conditioning relative to typical real acoustic scenes where such close pairs are rarer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SoundSculpt, an ambisonic-in-ambisonic-out UNet for target sound extraction from first-order ambisonic mixtures. The model is conditioned on a target direction (azimuth/elevation) and a semantic embedding derived from a text description of the target sound, using either a contrastively trained SoundWords encoder or a BERT encoder. The system is trained on a large synthetic dataset of image-source ambisonic RIRs mixed with source waveforms and on a semi-synthetic set of mixtures derived from the STARSS23 real recordings, where two independently recorded single-event clips are gain-scaled, randomly rotated, and mixed. Evaluations use SI-SDRi improvement against three signal-processing baselines. The central claim is that combined spatial and semantic conditioning outperforms spatial-only conditioning, especially when a secondary source is within 15 degrees of the target direction.
Significance. If the central claim holds, this is a useful contribution to target sound extraction in spatial audio, with practical relevance for VR/AR scenarios where a user can point at and describe a sound source. The paper offers a large-scale dataset (1.2M synthetic pairs plus 400k STARSS23-derived pairs), a clean architecture, and comparisons to multiple baselines. The synthetic experiments are self-consistent and show a consistent pattern of improvement from adding semantic conditioning. However, the real-data evidence is weakened by the artificial construction of the STARSS23 test mixtures and the absence of any uncertainty quantification. The work is a reasonable proof-of-concept, but the evaluation needs strengthening before the real-scene benefit can be considered established.
major comments (3)
- [Section 2.3, Table 1] The STARSS23 test mixtures are built from two independently recorded single-event clips with random gains and independent random rotations of each FOA segment. The target and secondary therefore do not share a common room impulse response, coherent reverberant field, or the natural level and onset statistics of simultaneously recorded events. This construction could systematically bias the comparison: in the only STARSS23 subset where spatial+SoundWords outperforms spatial-only (the close-secondary subset, 1.19 vs −0.59 dB), the independent mixing may make spatial cues less reliable and inflate the apparent benefit of semantic conditioning. Please evaluate on naturally co-occurring, simultaneously recorded sound events from the original STARSS23 evaluation set, or otherwise justify that the constructed mixtures preserve the properties on which the headline claim depends.
- [Table 1] No error bars, confidence intervals, or significance tests are reported for any of the SI-SDRi comparisons. Several differences used to support the conclusions are small (e.g., the 0.2 dB difference between semantic-only conditions on synthetic data and the 0.6 dB difference between spatial-only and spatial+SoundWords on the STARSS23 no-close subset). Without a measure of variance or paired testing across test segments, the claim that combined conditioning substantially outperforms spatial-only is not statistically supported. Please report bootstrap confidence intervals or paired significance tests for the key comparisons.
- [Section 2.3 and Section 3] The test construction imposes a 50% probability that a secondary source is within ±15° of the target direction, and this artificial prevalence shapes the 'all' columns of Table 1. If real ambisonic scenes have a lower rate of close secondary sources, the headline average benefit of semantic conditioning would be smaller; notably, the STARSS23 no-close subset shows spatial-only outperforming spatial+SoundWords by 0.6 dB. The authors should discuss how the 50% figure relates to real usage or provide a sensitivity analysis over this probability, and should temper the claim that combined conditioning is beneficial on realistic recordings.
minor comments (5)
- [Section 2.3] The sentence 'additionally, for both datasets, with 50% probability we placed a secondary source ... near the target source' is ambiguous because a secondary source is always present; clarify that the 50% refers to the probability that the secondary lies within ±15°.
- [Table 1] Condition names are inconsistently capitalized ('Bert' vs 'BERT', 'Soundwords' vs 'SoundWords'); use the same spelling throughout the paper.
- [Section 3] The statement that 'there are fewer secondary sources in the STARSS23 mixtures' is difficult to reconcile with the test construction, which always includes exactly one secondary source per mixture; please clarify what is meant.
- [Abstract and Section 2.3] The abstract refers to 'real ambisonic mixtures,' but the STARSS23 test inputs are constructed by mixing and rotating isolated clips; consider using a more precise term such as 'semi-synthetic' or 'real-source mixtures'.
- [References] Reference [14] is a web resource; please ensure it is permanently accessible and provide a standard citation format if possible.
Circularity Check
No significant circularity: SoundSculpt's headline results are held-out empirical comparisons, and the self-cited prior work is used only as design context.
full rationale
SoundSculpt is an empirical systems paper with no derivation chain in which a claimed prediction is defined as a fitted input. The central results in Table 1 are SI-SDRi scores computed on held-out test splits (the STARSS23 evaluation set and a separately generated synthetic test set) that are not used for training. The only tuned quantities are baseline parameters and the synthetic/STARSS23 training mix ratio, both explicitly acknowledged in Sections 2.4 and 2.5. The architecture draws inspiration from Kilgour et al. [7], and prior work [8] and [9] is cited as context for semantically conditioned extraction in mono and binaural settings; these self-citations are not used to establish the paper's quantitative claims. The constructed STARSS23 test mixtures, formed by rotating and mixing two independently recorded segments with a 50% probability of a within-15-degree secondary source, are an evaluation design choice. One may question their external validity, but the comparison remains an empirical measurement on held-out data rather than a circular reduction. No step satisfies the quoted-equation or self-citation test for circularity, so the score is 0.
Assumptions & free parameters
free parameters (2)
- Spherical cap spread for loudness baseline =
60 degrees
- Synthetic-to-STARSS23 batch mix ratio =
75% synthetic / 25% STARSS23
assumptions (5)
- domain assumption The target source direction is exactly known at test time (θs, ϕs are provided without noise)
- domain assumption The target source does not move during the 4.096s segment
- domain assumption Image-source simulation with shoebox rooms and frequency-dependent wall filters produces realistic ambisonic RIRs
- domain assumption Rotating real FOA segments with rotation matrices yields physically valid ambisonic recordings
- domain assumption The average SI-SDRi across the four FOA channels is a valid proxy for perceived extraction quality
Cite this review
Pith. "Pith review of SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction." pith.science (2026). https://pith.science/paper/DRMX3DBW
@misc{pith2026250600273,
author = {Pith},
title = {Pith review of: SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/DRMX3DBW}},
note = {Machine review of arXiv:2506.00273}
}
read the original abstract
This paper introduces SoundSculpt, a neural network designed to extract target sound fields from ambisonic recordings. SoundSculpt employs an ambisonic-in-ambisonic-out architecture and is conditioned on both spatial information (e.g., target direction obtained by pointing at an immersive video) and semantic embeddings (e.g., derived from image segmentation and captioning). Trained and evaluated on synthetic and real ambisonic mixtures, SoundSculpt demonstrates superior performance compared to various signal processing baselines. Our results further reveal that while spatial conditioning alone can be effective, the combination of spatial and semantic information is beneficial in scenarios where there are secondary sound sources spatially close to the target. Additionally, we compare two different semantic embeddings derived from a text description of the target sound using text encoders.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction In recent years, spatial audio has become increasingly popular due to the introduction of new VR/AR headsets. Ambisonics, a popular spatial audio format, enables binaural rendering of head-tracked spatial audio and is commonly used in VR [1]. Ambisonic signals can be recorded using spherical microphone arrays [2] or emerging techniques that e...
-
[2]
Method 2.1. Problem Setup Let a(t, θ, ϕ) be the plane wave amplitude distribution describ- ing an incident sound field. We denote the azimuth angle asθ ∈ [0, 2π] and the elevation angle as ϕ ∈ [− π 2 , π 2 ]. The ambisonic signal, x(t) = [x00(t), ..., xnm(t), ...xN N(t)]T ∈ R(N +1)2 , represents this sound field using a spherical harmonics (SH) ex- pansio...
-
[3]
Results and Discussion Results are presented in Table 1. We present SI-SDRi (averaged across channels) for the baseline algorithms and SoundSculpt models that use various types of conditioning. SI-SDRi is aver- aged across the respective datasets. We evaluate the algorithms on the test split of the STARSS23 mixtures and the synthetic dataset. In addition ...
-
[4]
SoundSculpt consis- tently outperformed traditional signal processing baselines
Conclusion This paper introduced SoundSculpt, a neural network model for target sound extraction from ambisonic recordings, leveraging both spatial and semantic conditioning. SoundSculpt consis- tently outperformed traditional signal processing baselines. The combination of spatial and semantic information proved bene- ficial in complex acoustic scenes wi...
-
[5]
Direction preserving wiener matrix filtering for ambisonic input-output systems,
A. Herzog and E. A. Habets, “Direction preserving wiener matrix filtering for ambisonic input-output systems,” in ICASSP 2019- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 446–450
work page 2019
-
[6]
SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction
investigate beamforming and matrix multi-channel Wiener filtering based methods to estimate a target ambisonic signal. However, if there is another sound source close to the desired target, these techniques might not perform effectively. This raises the question of whether information other than spatial direction can be employed as a conditioning factor f...
work page Pith review arXiv 2025
-
[7]
Text-driven separation of arbitrary sounds,
K. Kilgour, B. Gfeller, Q. Huang, A. Jansen, S. Wisdom, and M. Tagliasacchi, “Text-driven separation of arbitrary sounds,” arXiv preprint arXiv:2204.05738, 2022
arXiv 2022
-
[8]
F. Zotter and M. Frank, Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality. Springer Nature, 2019
work page 2019
Show all 34 references
-
[9]
Rafaely, Fundamentals of spherical array processing
B. Rafaely, Fundamentals of spherical array processing . Springer, 2015, vol. 8
2015
-
[10]
Ambisonics encoding for arbitrary microphone arrays incorpo- rating residual channels for binaural reproduction,
Y . Gayer, V . Tourbabin, Z. Ben-Hur, J. Donley, and B. Rafaely, “Ambisonics encoding for arbitrary microphone arrays incorpo- rating residual channels for binaural reproduction,”arXiv preprint arXiv:2402.17362, 2024
2024 arXiv
-
[11]
Spatial transformations for the en- hancement of ambisonic recordings,
M. Kronlachner and F. Zotter, “Spatial transformations for the en- hancement of ambisonic recordings,” in Proceedings of the 2nd International Conference on Spatial Audio, Erlangen, 2014
2014
-
[12]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2
2019
-
[13]
Direction and reverberation pre- serving noise reduction of ambisonics signals,
A. Herzog and E. A. P. Habets, “Direction and reverberation pre- serving noise reduction of ambisonics signals,”IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, vol. 28, pp. 2461–2475, 2020
2020
-
[14]
Bert experts (wikibooks),
Google, “Bert experts (wikibooks),” https://tfhub.dev/google/ experts/bert/wiki books/2, 2023, accessed: January 10, 2025
2023
-
[15]
Real-time target sound extraction,
B. Veluri, J. Chan, M. Itani, T. Chen, T. Yoshioka, and S. Gol- lakota, “Real-time target sound extraction,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5
2023
-
[16]
Se- mantic hearing: Programming acoustic scenes with binaural hear- ables,
B. Veluri, M. Itani, J. Chan, T. Yoshioka, and S. Gollakota, “Se- mantic hearing: Programming acoustic scenes with binaural hear- ables,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, pp. 1–15
2023
-
[17]
Binaural angular separation network,
Y . Yang, G. Sung, S.-F. Shih, H. Erdogan, C. Lee, and M. Grund- mann, “Binaural angular separation network,” in ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1201–1205
2024
-
[18]
Rezero: Region-customizable sound extrac- tion,
R. Gu and Y . Luo, “Rezero: Region-customizable sound extrac- tion,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
2024
-
[19]
On the modeling of rectangular geometries in room acoustic sim- ulations,
E. De Sena, N. Antonello, M. Moonen, and T. Van Waterschoot, “On the modeling of rectangular geometries in room acoustic sim- ulations,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 23, no. 4, pp. 774–786, 2015
2015
-
[20]
Efficient encoding and decoding of bin- aural sound with resonance audio,
M. Gorzel, A. Allen, I. Kelly, J. Kammerl, A. Gungormusler, H. Yeh, and F. Boland, “Efficient encoding and decoding of bin- aural sound with resonance audio,” in Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio. Audio E...
2019
-
[21]
Clotho: An audio cap- tioning dataset,
K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 736–740
2020
-
[22]
Film: Visual reasoning with a general conditioning layer,
E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Proceedings of the AAAI conference on artificial intelligence , vol. 32, no. 1, 2018
2018
-
[23]
Random source gains and si- lencing of secondary sources were applied to increase diversity
that are not part of FSD50k. Random source gains and si- lencing of secondary sources were applied to increase diversity. A similar process was used to create a synthetic test dataset of 120k pairs, with speech segments from LibriTTS dev-clean split [24] and arbitrary sounds f...
-
[24]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[25]
These descriptions were used to extract semantic embeddings for the target sounds
and a sound event classifier. These descriptions were used to extract semantic embeddings for the target sounds. An additional 400k training input/target pairs were created by sampling and mixing segments from the development set of the STARSS23 dataset [26]. This dataset cons...
-
[26]
Image method for efficiently sim- ulating small-room acoustics,
J. B. Allen and D. A. Berkley, “Image method for efficiently sim- ulating small-room acoustics,” The Journal of the Acoustical So- ciety of America, vol. 65, no. 4, pp. 943–950, 1979
1979
-
[27]
A note on a method for generating points uniformly on n-dimensional spheres,
M. E. Muller, “A note on a method for generating points uniformly on n-dimensional spheres,” Communications of the ACM , vol. 2, no. 4, pp. 19–20, 1959
1959
-
[28]
Libri-light: A benchmark for asr with limited or no su- pervision,
J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar´e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen et al., “Libri-light: A benchmark for asr with limited or no su- pervision,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Sig...
2020
-
[29]
Fsd50k: an open dataset of human-labeled sound events,
E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd50k: an open dataset of human-labeled sound events,” IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2021
2021
-
[30]
Freesound - Collaborative database of creative-commons li- censed sound effects,
“Freesound - Collaborative database of creative-commons li- censed sound effects,” https://freesound.org/, 2025, accessed: Jan- uary 10, 2025
2025
-
[31]
Libritts: A corpus derived from librispeech for text- to-speech,
H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,” arXiv preprint arXiv:1904.02882, 2019
1904 arXiv
-
[32]
Audio set: An ontology and human-labeled dataset for audio events,
J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE inter- national conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017,...
2017
-
[33]
Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,
K. Shimada, A. Politis, P. Sudarsanam, D. A. Krause, K. Uchida, S. Adavanne, A. Hakala, Y . Koyama, N. Takahashi, S. Takahashi et al., “Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,”Ad- vances in Neural ...
2024
-
[34]
Direction specific ambisonics source separation with end-to-end deep learning,
F. Lluis, N. Meyer-Kahlen, V . Chatziioannou, and A. Hofmann, “Direction specific ambisonics source separation with end-to-end deep learning,” Acta Acustica, vol. 7, p. 29, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.