RECORD
LiMuSE: Lightweight Multi-modal Speaker Extraction
Audio and Speech Processing · Submitted 7 Nov 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim The experiments on the MC_GRID dataset demonstrate that LiMuSE achieves on par or better performance with a much smaller number of parameters and less model complexity.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Multi-modal cues, including spatial information, facial expression and voiceprint, are introduced to the speech separation and speaker extraction tasks to serve as complementary information to achieve better performance. However, the introduction of these cues brings about an increasing number of parameters and model complexity, which makes it harder to deploy these models on resource-constrained devices. In this paper, we alleviate the aforementioned problem by proposing a Lightweight Multi-modal framework for Speaker Extraction (LiMuSE). We propose to use GC-equipped TCN, which incorporates Group Communication (GC) and Temporal Convolutional Network (TCN) in the Context Codec module, the audio block and the fusion block. The experiments on the MC_GRID dataset demonstrate that LiMuSE achieves on par or better performance with a much smaller number of parameters and less model complexity. We further investigate the impacts of the quantization of LiMuSE. Our code and dataset are provided.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Multi-modal cues, including spatial information, facial expression and voiceprint, are introduced to the speech separation and speaker extraction tasks to serve as complementary information to achieve better performance. | background 1.00 | 0.10 | 0.03 | asserted 1.00 |
| s1 | However, the introduction of these cues brings about an increasing number of parameters and model complexity, which makes it harder to deploy these models on resource-constrained devices. | background 0.81 | 0.12 | 0.02 | asserted 1.00 |
| s2 | In this paper, we alleviate the aforementioned problem by proposing a Lightweight Multi-modal framework for Speaker Extraction (LiMuSE). | contribution 0.99 | 0.84 | 0.10 | asserted 0.95 |
| s3 | We propose to use GC-equipped TCN, which incorporates Group Communication (GC) and Temporal Convolutional Network (TCN) in the Context Codec module, the audio block and the fusion block. | method 0.96 | 0.90 | 0.04 | asserted 0.93 |
| s4 | The experiments on the MC_GRID dataset demonstrate that LiMuSE achieves on par or better performance with a much smaller number of parameters and less model complexity. | result 1.00 | 0.96 | 0.82 | asserted 1.00 |
| s5 | We further investigate the impacts of the quantization of LiMuSE. | contribution 0.91 | 0.90 | 0.02 | unclear 0.61 |
| s6 | Our code and dataset are provided. | contribution 0.96 | 0.70 | 0.02 | asserted 0.99 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Multi-modal cues, including spatial information, facial expression and voiceprint, are introduced to the speech separation and speaker extraction tasks to serve as complementary information to achieve better performance. However, the introduction of these cues brings about an increasing number of parameters and model complexity, which makes it harder to deploy these models on resource-constrained devices. In this paper, we alleviate the aforementioned problem by proposing a Lightweight Multi-modal framework for Speaker Extraction (LiMuSE). We propose to use GC-equipped TCN, which incorporates Group Communication (GC) and Temporal Convolutional Network (TCN) in the Context Codec module, the audio block and the fusion block. The experiments on the MC_GRID dataset demonstrate that LiMuSE achieves on par or better performance with a much smaller number of parameters and less model complexity. We further investigate the impacts of the quantization of LiMuSE. Our code and dataset are provided.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites · 1 sentence without a confident label
Uses the method or tool of 2 papers
-
Then we use the SMS-WSJ toolkit [ref] to generate the simulated anechoic dual-channel audio mixture.
SMS-WSJ: Database, performance measures, and baseline recipe for multi-channel source separation and recognition · Datasets · 1.00
-
For activation quantization, we use the most commonly used min-max linear quantization [ref] , which is defined as: [equation] [equation] where $X$ is the tensor to be quantized, $x$ is the element in $X$, $X_min$ and $X_max$ are the minimum and maximum elements, respectively.
Quantizing deep convolutional networks for efficient inference: A whitepaper · Activation Quantization · 1.00
Cites as background 1 paper
-
Besides, various model compression methods have been proposed to lighten deep neural networks, such as parameter pruning [ref] , knowledge distillation [ref] and quantization [ref] .
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients · Introduction · 0.93
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.