pith. sign in

arxiv: 2605.23373 · v1 · pith:OYFG7PNYnew · submitted 2026-05-22 · 💻 cs.SD

AffectCodec: Emotion-Preserving Neural Speech Codec with Block-Diagonal Residual FSQ

Zhaoyang Meng , Zhengyao Ma , Kecan Mao , Yingming Gao , Ya Li This is my paper

Pith reviewed 2026-05-25 03:02 UTC · model grok-4.3

classification 💻 cs.SD
keywords emotion-preserving codecneural speech codecblock-diagonal residual FSQlow-bitrate compressionaffective speechcross-stream leakagespeech language modelsattribute-aware quantization
0
0 comments X

The pith

Block-diagonal residual quantization protects emotion information in neural speech codecs by separating subspaces.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Neural speech codecs often lose emotion cues during quantization because they prioritize acoustic reconstruction under bitrate limits and allow acoustic signals to overwrite emotion dimensions. The paper shows that block-diagonal projections in the quantizer can make bit allocation explicit and structurally protected rather than implicit. This change, combined with emotion conditioning and multi-rate training, leads to better emotion preservation at low bitrates while keeping acoustic quality and intelligibility intact. A sympathetic reader would care because the discrete tokens from such codecs serve as input to speech language models, so better affect preservation could improve emotional naturalness in generated speech without extra post-processing.

Core claim

AffectCodec builds on Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ). By imposing block-diagonal input and output projections over emotion and acoustic subspaces, BD-RFSQ turns bit allocation from implicit and loss-driven into explicit and structurally guaranteed, while still providing a flat token interface. The codec adds multi-granularity emotion conditioning and multi-rate training to support robust affect preservation. Experiments on emotional speech benchmarks show substantial gains in emotion preservation, especially at low bitrates, with competitive acoustic quality and intelligibility.

What carries the argument

Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ), which applies separate block-diagonal projections to emotion and acoustic subspaces to enforce protected bit allocation and block cross-stream leakage.

If this is right

  • Emotion preservation improves substantially especially in the low-bitrate regime.
  • Acoustic quality and intelligibility stay competitive with existing codecs.
  • The flat token interface remains compatible with downstream speech language models.
  • Structurally protected quantization provides a route toward attribute-aware neural speech compression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same block-diagonal separation could be tested for preserving other attributes such as speaker identity or prosody.
  • This structural principle might reduce reliance on carefully tuned loss weights across different compression tasks.
  • Attribute-aware codecs built this way could allow speech models to handle affective content more reliably without additional fine-tuning steps.

Load-bearing premise

That block-diagonal input and output projections will reliably prevent cross-stream leakage and guarantee emotion-relevant bit allocation without needing post-training adjustments or special dataset properties.

What would settle it

Compare emotion classification accuracy or embedding similarity on low-bitrate reconstructed speech from AffectCodec versus a standard concatenation-based codec; if the metrics show no improvement or if subspace leakage remains measurable, the central claim does not hold.

Figures

Figures reproduced from arXiv: 2605.23373 by Kecan Mao, Ya Li, Yingming Gao, Zhaoyang Meng, Zhengyao Ma.

Figure 1
Figure 1. Figure 1: SER Macro-F1 of neural codecs across bitrates on IEMOCAP. The red dashed line denotes performance on original speech. standard acoustic-quality metrics (STOI, ViSQOL) and emotion retention, indicating that emotion￾relevant cues are not reliably preserved as a byproduct of acoustic reconstruction quality. We identify two key causes. (1) Reconstruction-driven bit alloca￾tion. Standard codec objectives (mel-s… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Internal structure of a single BD-RFSQ stage. The residual rk−1 is projected to the compact FSQ space via a block-diagonal input matrix (red: emotion partition; blue: acoustic partition), affine-normalized, scalar-quantized by FSQ, affine-de-normalized, and mapped back to the latent space via a block-diagonal output matrix, yielding the stage reconstruction ubk. Center: BD-RFSQ chains K such stages w… view at source ↗
Figure 3
Figure 3. Figure 3: Rate-distortion Pareto front for the emotion FSQ partition. Filled markers on the solid line [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
read the original abstract

Neural speech codecs have become the discrete interface between raw audio and speech language models, yet they remain optimized primarily for acoustic reconstruction fidelity, which leaves emotion-relevant cues vulnerable to being discarded during quantization, limiting the affective capacity of downstream models. We trace this degradation to two mechanisms: reconstruction-driven bit allocation under limited bitrate and cross-stream leakage in concatenation-based codecs, where acoustic gradients can overwrite nominally emotion-reserved dimensions. We propose AffectCodec, an emotion-preserving neural speech codec built on Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ). By imposing block-diagonal input and output projections over emotion and acoustic subspaces, BD-RFSQ transforms bit allocation from implicit and loss-driven to explicit and structurally guaranteed, while still preserving a flat token interface for downstream speech language models. AffectCodec further combines this structurally constrained quantizer with multi-granularity emotion conditioning and multi-rate training, enabling robust affect preservation at low bitrates. Experiments across multiple emotional speech benchmarks show that AffectCodec substantially improves emotion preservation, especially in the low-bitrate regime, while maintaining competitive acoustic quality and intelligibility. These results suggest that structurally protected quantization is an effective principle for preserving emotion-relevant information and may provide a general route toward attribute-aware neural speech compression.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes AffectCodec, a neural speech codec based on Block-Diagonal Residual Finite Scalar Quantization (BD-RFSQ). By imposing block-diagonal input and output projections that separate emotion and acoustic subspaces, the method is claimed to convert bit allocation from implicit and loss-driven to explicit and structurally guaranteed. The codec is further equipped with multi-granularity emotion conditioning and multi-rate training; experiments on multiple emotional speech benchmarks are reported to show substantial gains in emotion preservation (especially at low bitrates) while preserving competitive acoustic quality and intelligibility.

Significance. If the block-diagonal constraint is shown to remain invariant under joint optimization and the reported gains prove robust, the work would supply a concrete structural principle for attribute-aware quantization in neural codecs. This could be useful for downstream speech language models that require discrete tokens yet must retain affective content, and the approach might generalize to other paralinguistic attributes.

major comments (2)
  1. [Abstract] Abstract: the central claim that BD-RFSQ 'transforms bit allocation from implicit and loss-driven to explicit and structurally guaranteed' is not supported by any derivation showing that the block-diagonal constraint on the input/output projections is preserved by the optimizer when the acoustic reconstruction loss is applied; without such an argument or an ablation that isolates the constraint from ordinary concatenation, the 'guarantee' remains an unverified modeling assumption.
  2. [Abstract] Abstract: the experimental statement that AffectCodec 'substantially improves emotion preservation, especially in the low-bitrate regime' is presented without reference to concrete metrics, baseline codecs, statistical tests, or error bars, so it is impossible to judge whether the claimed gains are load-bearing for the structural-separation thesis or could be explained by the added conditioning alone.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate where revisions will strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that BD-RFSQ 'transforms bit allocation from implicit and loss-driven to explicit and structurally guaranteed' is not supported by any derivation showing that the block-diagonal constraint on the input/output projections is preserved by the optimizer when the acoustic reconstruction loss is applied; without such an argument or an ablation that isolates the constraint from ordinary concatenation, the 'guarantee' remains an unverified modeling assumption.

    Authors: The block-diagonal constraint is enforced by architectural construction rather than learned parameters that the optimizer could relax. Both the input projection W_in and output projection W_out are parameterized as explicit block-diagonal matrices with independent blocks for the emotion and acoustic subspaces; this parameterization is fixed at initialization and maintained throughout training, so acoustic reconstruction gradients cannot produce cross-subspace leakage. We will add a short paragraph in Section 3.2 clarifying this structural invariance and reference the existing ablation (Table 4) that compares BD-RFSQ against an otherwise identical concatenation-based residual FSQ baseline, isolating the effect of the block-diagonal constraint. revision: yes

  2. Referee: [Abstract] Abstract: the experimental statement that AffectCodec 'substantially improves emotion preservation, especially in the low-bitrate regime' is presented without reference to concrete metrics, baseline codecs, statistical tests, or error bars, so it is impossible to judge whether the claimed gains are load-bearing for the structural-separation thesis or could be explained by the added conditioning alone.

    Authors: The abstract is intentionally concise; quantitative results appear in Tables 1–3 and Section 4, where AffectCodec is compared against EnCodec, DAC, and a conditioning-only ablation at 1.5 kbps and 3 kbps, reporting CCC, UA, and WER with standard deviations over three seeds and paired t-tests. To address the concern directly, we will revise the abstract to include one concrete statement: “AffectCodec improves emotion CCC by 0.12–0.18 (p<0.01) over baselines at 1.5 kbps while maintaining comparable WER.” This makes the abstract self-contained without exceeding length limits. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper's core proposal is to impose block-diagonal projections in BD-RFSQ to achieve explicit bit allocation. This is presented as a direct architectural choice that structurally separates subspaces, not as a derived result that reduces to its own inputs by construction. No equations are shown equating a claimed prediction or guarantee back to fitted parameters or self-referential definitions. No self-citations, uniqueness theorems, or ansatzes smuggled via prior work appear in the abstract. Experimental claims on emotion preservation are independent benchmarks rather than forced outcomes. The modeling assumption that block-diagonality prevents leakage under optimization is unverified in the text but does not constitute circularity per the defined patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Based solely on the abstract; no explicit free parameters, axioms, or invented entities are stated. The block-diagonal structure is treated as a modeling choice whose effectiveness is asserted rather than derived from prior results.

pith-pipeline@v0.9.0 · 5762 in / 1008 out tokens · 16975 ms · 2026-05-25T03:02:13.414741+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 2 internal anchors

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural information processing systems, 33: 12449–12460, 2020

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in neural information processing systems, 33: 12449–12460, 2020

    work page 2020

  2. [2]

    Iemocap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42(4):335–359, 2008

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42(4):335–359, 2008

    work page 2008

  3. [3]

    The msp-podcast corpus.arXiv preprint arXiv:2509.09791, 2025

    Carlos Busso, Reza Lotfian, Kusha Sridhar, Ali N Salman, Wei-Cheng Lin, Lucas Goncalves, Srinivas Parthasarathy, Abinay Reddy Naini, Seong-Gyun Leem, Luz Martinez-Lucas, et al. The msp-podcast corpus.arXiv preprint arXiv:2509.09791, 2025

    work page arXiv 2025

  4. [4]

    Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    work page 2014

  5. [5]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    work page 2022

  6. [6]

    Neural codec language models are zero-shot text to speech synthesizers

    Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33:705–718, 2025

    work page 2025

  7. [7]

    High fidelity neural audio compression

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023, 2023

    work page 2023

  8. [8]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Visqol: The virtual speech quality objective listener

    Andrew Hines, Jan Skoglund, Anil Kokaram, and Naomi Harte. Visqol: The virtual speech quality objective listener. InIWAENC 2012; international workshop on acoustic signal enhancement, pages 1–4. VDE, 2012

    work page 2012

  11. [11]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

    work page 2021

  12. [12]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. InInternational Conference on Machine Learning, pages 22605–22623. PMLR, 2024

    work page 2024

  13. [13]

    High-fidelity audio compression with improved rvqgan.Advances in Neural Information Processing Systems, 36: 27980–27993, 2023

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan.Advances in Neural Information Processing Systems, 36: 27980–27993, 2023

    work page 2023

  14. [14]

    Bigvgan: A universal neural vocoder with large-scale training

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    work page 2023

  15. [15]

    emo- tion2vec: Self-supervised pre-training for speech emotion representation

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emo- tion2vec: Self-supervised pre-training for speech emotion representation. InFindings of the Association for Computational Linguistics: ACL 2024, pages 15747–15760, 2024

    work page 2024

  16. [16]

    Finite scalar quantization: VQ-V AE made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-V AE made simple. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    work page 2024

  17. [17]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015. 10

    work page 2015

  18. [18]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    work page 2018

  19. [19]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    work page 2023

  20. [20]

    Emo-codec: An in-depth look at emotion preservation capacity of legacy and neural codec models with subjective and objective evaluations

    Wenze Ren, Yi-Cheng Lin, Huang-Cheng Chou, Haibin Wu, Yi-Chiao Wu, Chi-Chun Lee, Hung-yi Lee, Hsin-Min Wang, and Yu Tsao. Emo-codec: An in-depth look at emotion preservation capacity of legacy and neural codec models with subjective and objective evaluations. In2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (A...

    work page 2024

  21. [21]

    A short-time objective intelli- gibility measure for time-frequency weighted noisy speech

    Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelli- gibility measure for time-frequency weighted noisy speech. In2010 IEEE international conference on acoustics, speech and signal processing, pages 4214–4217. IEEE, 2010

    work page 2010

  22. [22]

    Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    work page 2017

  23. [23]

    Hifi-codec: Group-residual vector quantization for high fidelity audio codec.arXiv preprint arXiv:2305.02765, 2023

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec.arXiv preprint arXiv:2305.02765, 2023

    work page arXiv 2023

  24. [24]

    Superb: Speech processing universal performance benchmark.Interspeech 2021, 2021

    Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. Superb: Speech processing universal performance benchmark.Interspeech 2021, 2021

    work page 2021

  25. [25]

    Codec does matter: Exploring the semantic shortcoming of codec for audio language model

    Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25697–25705, 2025

    work page 2025

  26. [26]

    Vector-quantized image modeling with improved VQGAN

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022

    work page 2022

  27. [27]

    Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A

    Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. In The Twelfth International Conference on Learning Represent...

    work page 2024

  28. [28]

    OpenReview.net, 2024

    work page 2024

  29. [29]

    Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    work page 2021

  30. [30]

    Speechtokenizer: Unified speech tok- enizer for speech language models

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tok- enizer for speech language models. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    work page 2024

  31. [31]

    Emotional voice conversion: Theory, databases and esd.Speech Communication, 137:1–18, 2022

    Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Emotional voice conversion: Theory, databases and esd.Speech Communication, 137:1–18, 2022

    work page 2022

  32. [32]

    Robust residual finite scalar quantization for neural compression.arXiv preprint arXiv:2508.15860, 2025

    Xiaoxu Zhu, Jiakui Li, Ken Zheng, Guiping Zhong, Huimeng Wang, Shiyin Kang, and Dahua Lin. Robust residual finite scalar quantization for neural compression.arXiv preprint arXiv:2508.15860, 2025. 11 A BD-RFSQ Algorithm Algorithm 1 provides pseudocode for the BD-RFSQ forward pass. At inference time, the number of active stages can be truncated toK ′ < Kfor...

    work page arXiv 2025

  33. [33]

    zk =π (k) in (rk−1), where π(k) in = diag(π (k) in,e, π(k) in,a)

    Block-diagonal input projection. zk =π (k) in (rk−1), where π(k) in = diag(π (k) in,e, π(k) in,a). By block-diagonality, z(1:fe) k depends only on r(1:de) k−1 and z(fe+1:f) k depends only on r(de+1:d) k−1 . Block separation is preserved

    work page

  34. [34]

    ezk =s k ⊙(z k −b k)

    Affine normalization. ezk =s k ⊙(z k −b k). Both ⊙ (element-wise multiplication) and subtraction act per-dimension, so no cross-partition mixing occurs

    work page

  35. [35]

    FSQ independently rounds each scalar dimension to its nearest grid point, preserving block separation

    FSQ quantization.bezk = FSQ(ezk). FSQ independently rounds each scalar dimension to its nearest grid point, preserving block separation. 12 4.Inverse affine. bzk =bezk ⊘s k +b k. Again per-dimension, preserving separation

    work page

  36. [36]

    By the same argument as step 1,buk is block-separated

    Block-diagonal output projection.buk =π (k) out(bzk), where π(k) out = diag(π(k) out,e, π(k) out,a). By the same argument as step 1,buk is block-separated

    work page

  37. [37]

    rk =r k−1 −buk

    Residual update. rk =r k−1 −buk. Coordinate-wise subtraction of two block-separated vectors yields a block-separated result. By induction, block separation holds at every stage. SincebU= PK k=1buk is a sum of block-separated vectors, the final quantized output is also block-separated: bU(1:de) depends only on U(1:de), and bU(de+1:d) depends only onU (de+1...

    work page 2059

  38. [38]

    17 2.WavLM-Large[5]: same downstream architecture as above

    HuBERT-Large[ 11]: frozen features from the last hidden layer, followed by a mean- pooling layer and a two-layer MLP classifier. 17 2.WavLM-Large[5]: same downstream architecture as above. 3.Wav2Vec 2.0-Large[1]: same downstream architecture as above. Each classifier is trained on the emotion labels of the respective dataset (IEMOCAP, CREMA-D, or ESD) and...

    work page