RECORD
Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables
Sound · Submitted 1 Nov 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim In-the-wild evaluation with participants in previously unseen indoor and outdoor scenarios shows that our proof-of-concept system can extract the target sounds and generalize to preserve the spatial cues in its binaural output.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Imagine being able to listen to the birds chirping in a park without hearing the chatter from other hikers, or being able to block out traffic noise on a busy street while still being able to hear emergency sirens and car honks. We introduce semantic hearing, a novel capability for hearable devices that enables them to, in real-time, focus on, or ignore, specific sounds from real-world environments, while also preserving the spatial cues. To achieve this, we make two technical contributions: 1) we present the first neural network that can achieve binaural target sound extraction in the presence of interfering sounds and background noise, and 2) we design a training methodology that allows our system to generalize to real-world use. Results show that our system can operate with 20 sound classes and that our transformer-based network has a runtime of 6.56 ms on a connected smartphone. In-the-wild evaluation with participants in previously unseen indoor and outdoor scenarios shows that our proof-of-concept system can extract the target sounds and generalize to preserve the spatial cues in its binaural output. Project page with code: https://semantichearing.cs.washington.edu
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Imagine being able to listen to the birds chirping in a park without hearing the chatter from other hikers, or being able to block out traffic noise on a busy street while still being able to hear emergency sirens and car honks. | background 0.99 | 0.04 | 0.03 | speculative 0.45 |
| s1 | We introduce semantic hearing, a novel capability for hearable devices that enables them to, in real-time, focus on, or ignore, specific sounds from real-world environments, while also preserving the spatial cues. | contribution 1.00 | 0.70 | 0.30 | asserted 1.00 |
| s2 | To achieve this, we make two technical contributions: 1) we present the first neural network that can achieve binaural target sound extraction in the presence of interfering sounds and background noise, and 2) we design a training methodology that allows our system to generalize to real-world use. | contribution 0.99 | 0.96 | 0.07 | asserted 1.00 |
| s3 | Results show that our system can operate with 20 sound classes and that our transformer-based network has a runtime of 6.56 ms on a connected smartphone. | result 1.00 | 0.97 | 0.09 | asserted 1.00 |
| s4 | In-the-wild evaluation with participants in previously unseen indoor and outdoor scenarios shows that our proof-of-concept system can extract the target sounds and generalize to preserve the spatial cues in its binaural output. | result 1.00 | 0.96 | 0.56 | asserted 0.91 |
| s5 | Project page with code: https://semantichearing.cs.washington.edu | future work 0.47 | 0.25 | 0.02 | unclear 0.83 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Imagine being able to listen to the birds chirping in a park without hearing the chatter from other hikers, or being able to block out traffic noise on a busy street while still being able to hear emergency sirens and car honks. We introduce semantic hearing, a novel capability for hearable devices that enables them to, in real-time, focus on, or ignore, specific sounds from real-world environments, while also preserving the spatial cues. To achieve this, we make two technical contributions: 1) we present the first neural network that can achieve binaural target sound extraction in the presence of interfering sounds and background noise, and 2) we design a training methodology that allows our system to generalize to real-world use. Results show that our system can operate with 20 sound classes and that our transformer-based network has a runtime of 6.56 ms on a connected smartphone. In-the-wild evaluation with participants in previously unseen indoor and outdoor scenarios shows that our proof-of-concept system can extract the target sounds and generalize to preserve the spatial cues in its binaural output. Project page with code: https://semantichearing.cs.washington.edu
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 1 sentence without a confident label
Uses the method or tool of 4 papers
-
Following [ref] , the spatial cue accuracy is measured using the difference in the interaural time differences (ITDs) and interaural level differences (ILDs) between the output binaural signal and the ground-truth binaural signal, denoted as $Δ$ITD and $Δ$ILD.
Real-time binaural speech separation with preserved spatial cues · Benchmarking the neural network · 1.00
-
In this work, for efficient on-device inference, we implemented the dynamic programming algorithm proposed in Fast Wavenet [ref] .
Fast Wavenet Generation Algorithm · Mask estimation network · 1.00
-
To mitigate this inefficiency, while achieving a large receptive field, our mask estimation network implements a Wavenet [ref] style dilated causal convolutions for processing the input and previous chunks.
WaveNet: A Generative Model for Raw Audio · Mask estimation network · 0.92
Uses data or values from 1 paper
-
So, we combined audio samples from four different datasets: FSD50K [ref] (general-purpose), ESC-50 [ref] (environmental sounds), MUSDB18 [ref] (music and vocals) and noise files for the DISCO [ref] dataset (noise sounds).
FSD50K: An Open Dataset of Human-Labeled Sound Events · Audio dataset curation. · 1.00
Compares with 2 papers
-
In contrast to more complex binaural extraction frameworks proposed specifically for speech, where each channel is separately and parallely processed [ref] , our design jointly processes the two channels for computational efficiency.
End-to-End Multi-Channel Speech Separation · High-level framework · 0.92
-
In contrast to more complex binaural extraction frameworks proposed specifically for speech, where each channel is separately and parallely processed [ref] , our design jointly processes the two channels for computational efficiency.
Real-time binaural speech separation with preserved spatial cues · High-level framework · 0.92
Cites as background 2 papers
-
However recent works have proposed neural networks that can achieve target sound extraction where clues about the target sound are provided either via audio [ref] , images [ref] , text [ref] , onomatopoeic words [ref] , or a one-hot vectors [ref] .
Listen to What You Want: Neural Network-based Universal Sound Selector · Background and Related Work · 0.88
-
Prior work has explored the use of sounds to perform activity recognition for wearables and smart home applications [ref] .
Sound Event Detection Guided by Semantic Contexts of Scenes · Background and Related Work · 1.00
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.