Pith. sign in

REVIEW 3 major objections 6 minor 48 references

SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single masked language model on RVQ audio tokens serves as both the sender's entropy model and the receiver's packet-loss concealer, and the resulting SoundSpring transceiver outperforms traditional and neural audio transmission…

desk verdict A genuinely neat dual-use of one masked language model for both audio compression and packet-loss concealment, backed by broad experiments; the main soft spot is the hand-designed training masking schedule, which deserves a sensitivity analysis but does not sink the paper. read the letter →

arxiv 2501.12696 v1 pith:Q3GBXCK5 submitted 2025-01-22 eess.AS cs.SDeess.SP

Shengshi Yao , Jincheng Dai , Xiaoqi Qin , Sixian Wang , Siye Wang , Kai Niu , Ping Zhang This is my paper · ORCID
classification eess.AScs.SDeess.SP
keywords audiotransceiverpacketlossconcealmentmaskedlanguagemodelneuralcodecresidualvectorquantizationentropycodingjointsource-channelreal-timecommunication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SoundSpring is an audio transceiver that uses one masked language model for two jobs: at the sender it estimates the probability of residual vector quantizer (RVQ) audio tokens so they can be entropy-coded more compactly, and at the receiver it predicts tokens lost or invalidated by packet loss. The paper's central claim is that these two roles are the same contextual prediction task, so a single model trained with a random masking schedule can compress audio and conceal arbitrary loss patterns without retransmission. The authors report that SoundSpring outperforms traditional codecs (Opus, AMR-WB, AAC) and neural baselines (SoundStream with and without FD-PLC) on speech and music under random and WLAN packet-loss channels, in both objective metrics and subjective MUSHRA ratings. The significance, if true, is that an LLM-style audio model can replace the separate error-control machinery of a layered digital transceiver while remaining compatible with existing physical-layer transmission.

What carries the argument

The central object is the dual-functional masked language model (MLM) acting on a two-dimensional grid of RVQ audio tokens, where the horizontal axis is time and the vertical axis is the residual quantizer layer. Tokens are sliced into coarse and fine groups, grouped into slices and groups of slices (GoS), and the sender conditions fine tokens on coarse tokens and key slices under a fixed dependency matrix $\Phi$; the receiver masks whatever tokens are lost or invalid in a concealment window and predicts them under a bi-directional dependency pattern $\Psi$. The unifying mechanism is the training masking schedule: with $\beta(\tau)=\frac{1}{2}(1+\cos(\tau\pi))$, the model sees both small masking ratios, matching concealment, and large masking ratios, matching entropy modeling, and randomly masks from layer $k$ to $K$, which makes the same Transformer handle multiple coding rates and arbitrary loss patterns. An auxiliary FEC layer protecting the coarse tokens is added to keep the foundation tokens safe, with MLM concealment handling what FEC cannot.

What would settle it

A concrete falsifying experiment: take a pre-trained SoundSpring model and run it on a channel whose loss process produces bursts far longer than any contiguous masked run in the training schedule, such as 50 consecutive lost frames, then compare PLCMOS and PESQ against Opus with LBRR at the same rate; if SoundSpring's quality falls below the traditional codec while its in-distribution performance was superior, the claim that one MLM generalizes to arbitrary packet-loss patterns is false.

Watch

Extended reading notes

Core claim

The discovery is that a bi-directional Transformer trained as a masked language model over RVQ audio tokens is simultaneously a compression engine and a packet-loss concealment engine. On the sender side, the model supplies the contextual probability mass functions needed for entropy coding of fine tokens, cutting the required codelength for an exemplar fine-layer token from 10 bits to 3.83 bits. On the receiver side, the same model, with lost tokens replaced by [MASK], predicts what was lost from the surviving neighbours, with a concealment dependency pattern that is bi-directional in time and causal along the RVQ layer axis. Because the training objective is a single log-loss over masked tokens, one model adapts to the fixed coding dependency pattern $\Phi$ used at the sender and to the diverse concealment dependency patterns $\Psi$ that arise from arbitrary packet losses. The paper argues this collapses the traditional divide between source coding and error control.

Load-bearing premise

The load-bearing premise is that the random masking schedule used in training—sometimes hiding many frames, sometimes few, with randomly chosen audio-token layers—covers the packet-loss patterns that real channels produce, including long bursts, and that every lost or corrupted packet is detected and treated as a masked token; if a channel produces losses that look nothing like the training masks, the same model will not both compress and conceal.

Editorial extensions

If this is right

  • One unified log-loss objective trains both the compressor's entropy model and the receiver's concealer, so improvements to contextual prediction directly improve both ends of the transceiver.
  • Variable-rate coding needs no extra models: adjusting the number of encoded RVQ layers $K$ changes the bitrate while the same MLM adapts, enabling rate control per channel state.
  • Under increasing packet loss, SoundSpring degrades gracefully like a joint source-channel code rather than suffering a cliff, because the MLM conceals progressively more lost tokens.
  • The streaming variant SoundSpring-S keeps real-time operation, with real-time factors below 1 on CPU, so the scheme is compatible with RTC-style latency constraints.
  • Coarse-token FEC plus MLM concealment forms a dual protection mechanism: FEC preserves the foundation tokens on which all fine tokens depend, and the MLM handles burst losses where redundant codes are themselves lost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated implication is that the random masking schedule is a form of data augmentation over loss patterns; the same recipe could be ported to any codec that produces RVQ-style discrete tokens, such as image or video tokenizers, yielding a generic compress-and-conceal transformer.
  • If the masking schedule is the only thing standing between training and a given channel, then matching the schedule to measured channel statistics, such as burst length or loss autocorrelation, should further improve concealment; this is a testable extension the paper does not run.
  • The strongest untested boundary is long correlated bursts: the paper evaluates WLAN traces, but a stress test with bursts far longer than any in training would reveal whether the MLM generalizes or merely interpolates between masking ratios.
  • Because the entropy model and concealment share weights, a channel that flips tokens rather than erasing them could probe whether the same model also detects corruption, not just conceals erasures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes SoundSpring, a packet-loss-resilient audio transceiver that combines a neural audio codec with residual vector quantization (RVQ) and a bidirectional masked language model (MLM). The same MLM is used at the transmitter as an entropy model for coding RVQ tokens and at the receiver to predict lost or invalid tokens for packet loss concealment. The method is instantiated in non-streaming and streaming (SoundSpring-S) variants, trained with a random frame-masking schedule, and evaluated on LibriSpeech and MTG-Jamendo under i.i.d. and WLAN burst-loss channels. The reported results show consistent improvements over Opus, AMR-WB, AAC, SoundStream, and SoundStream+FD-PLC in SDR, SI-SNR, PESQ, PLCMOS, ViSQOL, and MUSHRA scores, with real-time factors below 1 on a CPU.

Significance. If the empirical claims hold, SoundSpring is a useful contribution: it demonstrates that one MLM can serve both compression and concealment, offering a digital, layered-architecture-compatible alternative to analog JSCC. The paper has several strengths: a broad set of objective and subjective metrics, two datasets, both i.i.d. and Markov-modeled WLAN loss traces, latency measurements, and a public code link. The main caveat is that the central claim rests on the generalization of a single random masking schedule to deployment loss patterns; the paper does not provide a sensitivity analysis or a matched/unmatched training control. The contribution is therefore promising but not yet fully established.

major comments (3)
  1. [Sec. IV-D.2, Eq. (3), Figs. 6-7] The training masking schedule is the load-bearing link between the sender and receiver roles of the MLM, but its coverage of deployment loss patterns is not established. In the non-streaming configuration, Eq. (3) defines each slice as a strided subset of frames, so losing one packet removes a periodic comb of frames; training, by contrast, masks entire randomly chosen frames from layer k to K. Moreover, beta(tau)=0.5(1+cos(tau*pi)) is U-shaped and is not derived from the Bernoulli or three-state Markov loss models used in the experiments. The paper reports no sensitivity analysis over the masking schedule and no control experiment showing that the chosen schedule, rather than another schedule, produces the reported gains. Please add ablations that train with masks matched to slice-level loss (non-streaming) and frame-level loss (streaming), and report the distribution of effective loss patterns seen in training versus test. Without this, the claim in Section I that the random masking strategy can handle 'arbitrary unpredictable packet loss patterns' is overreaching.
  2. [Table I, Sec. V-B.2] The bitrate values labeled 'MLM' in Table I are not defined. Please state whether they are actual arithmetic-coded bitstream lengths or a cross-entropy lower bound computed from the MLM PMFs. The compression-efficiency claim is central, and a theoretical cross-entropy number would be optimistic relative to a real entropy coder; if actual lengths are used, specify the entropy coder, the packetization overhead, and how FEC redundancy is included in the reported total bitrate (e.g., the precise contribution of the coarse-token FEC to the 5.0 kbps figure in Section V-B.3).
  3. [Sec. III-B, Sec. IV-D.2] The paper describes the sender-side objective as minimizing cross-entropy under the agreed coding dependency pattern Phi (Fig. 4), but training uses the random masking schedule of Section IV-D.2 rather than masks drawn from Phi. Please clarify how the random schedule specializes to the exact conditioning pattern used for entropy coding, or provide a control experiment that trains the sender with masks drawn exactly from Phi and reports the resulting bitrate and concealment metrics. This is needed to attribute the compression gain to the proposed dual-functional masking strategy.
minor comments (6)
  1. [Abstract] The abstract contains the typo 'casual-order'; it should be 'causal-order'. The phrase 'serve dual-functionalities' should also be corrected to 'serves dual functionalities'.
  2. [Figs. 10 and 12] The legend labels in Figs. 10 and 12 appear garbled (e.g., 'SoundStrea', 'O us', and Unicode replacement characters in Fig. 12). Ensure the final PDF renders all axis labels and legend entries correctly.
  3. [Sec. IV-D.2] Please define the sequence length T used in training and state explicitly how the continuous masking ratio beta(tau) is converted to an integer number of masked frames, including the handling of the floor operation.
  4. [Sec. V-B.3] The description 'FEC redundancy of coarse tokens is added to the subsequent frame' is underspecified; please report the FEC code, code rate, and temporal interleaving used for the coarse-token protection.
  5. [Sec. V-C, Fig. 13] The MUSHRA results are reported without confidence intervals or significance tests. Since several comparisons in Fig. 13 are close, add per-condition 95% confidence intervals and a paired significance test (e.g., Wilcoxon signed-rank) to support the subjective-quality claims.
  6. [Eq. (6)] The definition of the MFCC distance is unclear: 'F is the number of mel scales (set to 4) with coefficient numbers of [8, 16, 32, 64]' should be rewritten to specify whether F denotes the number of filterbanks or the total number of coefficients, and how the vector m_f is formed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the central claims are empirical and self-contained.

full rationale

SoundSpring's central claims are empirical rather than derivational. The masked language model outputs a contextual PMF p(z_t,k | M) that is used for entropy coding at the sender (Section III-B) and for token concealment at the receiver (Section III-C), with both objectives trained as a unified log-loss over mask tokens (Section IV-D.2). This is a shared architectural design, not a case where a predicted quantity is defined in terms of a fitted quantity. The compression gain is measured by actually entropy-coding the tokens with the MLM's PMF against a uniform prior and against SoundStream without the MLM (Table I, Fig. 10), and the concealment quality is evaluated on held-out speech and music under i.i.d. and WLAN packet-loss traces (Figs. 10-13), so the reported 'predictions' are not forced by construction. The masking schedule beta(tau)=0.5(1+cos(tau*pi)) is an empirically chosen training distribution; whether it generalizes to deployment loss patterns is a robustness assumption, not a circularity. The self-citations ([7], [8], [22]) are contextual prior work and are not load-bearing for the main result, which is also supported by the independent reference [21] for the language-modeling-as-compression principle. No fitted parameter is renamed as a prediction, and no uniqueness or exclusivity claim is imported from the authors' own prior work. The paper is therefore best assessed as empirically argued and not circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the RVQ token structure, the assumption that random masking training generalizes to arbitrary packet loss patterns, and packet-level loss detection. The system also depends on hand-set architecture hyperparameters (layer split, slice geometry, masking schedule) that are not derived from first principles.

free parameters (3)
  • Coarse-layer count N_C = 3 (16 kHz speech), 2 (48 kHz music)
    Hand-chosen division of RVQ layers into coarse and fine tokens; this split determines the entropy modeling scope and error propagation behavior and is set without a principled criterion.
  • Slice scheduling parameters (S, T_G, T_C, T_S, T_F) = S=10, T_G=150 frames (music), T_C=15/30, T_S=3 frames, T_F=2 (streaming)
    These govern context length, packetization, and latency; the paper explores a few values in Table II-III but does not optimize them systematically.
  • Masking schedule beta(tau)=0.5(1+cos(tau*pi)) = Mapping from tau in [0,1] to masking ratio in (0,1)
    Hand-designed distribution over frame masking ratios; central to training the dual-functional MLM, but no evidence it is matched to real channel loss distributions beyond the tested traces.
assumptions (4)
  • standard math Conditional entropy is no larger than unconditional entropy, so modeling inter-token dependencies improves coding efficiency.
    Invoked in Section III-B to justify the entropy model; standard information theory.
  • domain assumption The residual vector quantizer (RVQ) tokens are causally dependent along the layer order (higher layers refine lower layers), and this dependency can be exploited without unacceptable error propagation.
    Assumed throughout Sections III-B and IV; the error propagation behavior along RVQ layers is treated as given.
  • domain assumption The MLM trained with the random masking schedule generalizes to arbitrary packet-loss patterns, including long burst losses.
    This is the core empirical bet of the paper; Section IV-D.2 defines the masking schedule, and the WLAN trace experiments test it but do not prove generalizability.
  • domain assumption Packet loss is detectable at the receiver via channel decoding failure or erasure state; corrupted packets are discarded and treated as lost.
    Section III assumes residual errors lead to undecodable bitstreams; without reliable packet-level error detection, the concealment logic fails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling." pith.science (2026). https://pith.science/paper/Q3GBXCK5

@misc{pith2026250112696,
  author       = {Pith},
  title        = {Pith review of: SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3GBXCK5}},
  note         = {Machine review of arXiv:2501.12696}
}
read the original abstract

In this paper, we propose "SoundSpring", a cutting-edge error-resilient audio transceiver that marries the robustness benefits of joint source-channel coding (JSCC) while also being compatible with current digital communication systems. Unlike recent deep JSCC transceivers, which learn to directly map audio signals to analog channel-input symbols via neural networks, our SoundSpring adopts the layered architecture that delineates audio compression from digital coded transmission, but it sufficiently exploits the impressive in-context predictive capabilities of large language (foundation) models. Integrated with the casual-order mask learning strategy, our single model operates on the latent feature domain and serve dual-functionalities: as efficient audio compressors at the transmitter and as effective mechanisms for packet loss concealment at the receiver. By jointly optimizing towards both audio compression efficiency and transmission error resiliency, we show that mask-learned language models are indeed powerful contextual predictors, and our dual-functional compression and concealment framework offers fresh perspectives on the application of foundation language models in audio communication. Through extensive experimental evaluations, we establish that SoundSpring apparently outperforms contemporary audio transmission systems in terms of signal fidelity metrics and perceptual quality scores. These new findings not only advocate for the practical deployment of SoundSpring in learning-based audio communication systems but also inspire the development of future audio semantic transceivers.

Figures

Figures reproduced from arXiv: 2501.12696 by the authors.

Figure 1
Figure 1. Overview of the architecture of SoundSpring. The audio waveform is mapped to audio latent features and then multiple sequences of audio tokens are [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of MLM in SoundSpring transceiver. The RVQ audio tokens [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Two-dimensional audio token grid of zt,k and audio token slice grid Sℓ,j . Every TG frames of tokens constitute a group of slices. The audio tokens generated by RVQ in bottom layers are assigned as coarse tokens, while the ones on the top layers are categorized into fine tokens. Key slices Key slices Coarse Key Others Fine slices Coarse slices 关键切片 关键切片 粗粒度 关键 其他 细粒度 切片 粗粒度切片 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: General dependency link of audio tokens and slice-wise dependency [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Possible received token grid patterns consisting of [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Concealing dependency matrix Ψ. The dependency is bi-directional along the time axis and causal along the layer axis. In practical, the masking pattern is obtained according to the state of tokens in the concealment window. is log2 M = 10 bits, with uniform prior assum…
Figure 8
Figure 8. Figure 8: Spectrum visualization demo of the spectrum of reconstructed audio [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Exemplar diagram of audio token processing for dual-functional MLM [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Quality versus loss ratio which are reported in objective distortion metrics and subjective quality scores over channels with i.i.d. random packet loss [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Perceptual qualities of real-time speech transmission over variable-loss channels. Results in the top row (a)-(b) are tested with consistent [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Cumulative distribution function (CDF) of audio quality metrics over a WLAN packet loss channel. The same [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: MUSHRA scores simulated over channels with (a) 10% random packet loss, (b) 30% random packet loss, (c) WLAN packet loss trace. We provide [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 41 canonical work pages

  1. [1]

    The adaptive multirate wide- band speech codec (AMR-WB),

    B. Bessette, R. Salami, R. Lefebvre, M. Jelinek, J. Rotola-Pukkila, J. Vainio, H. Mikkola, and K. Jarvinen, “The adaptive multirate wide- band speech codec (AMR-WB),” IEEE Transactions on Speech and Audio Processing, vol. 10, no. 8, pp. 620–636, 2002

    work page 2002

  2. [2]

    Definition of the Opus audio codec,

    J.-M. Valin, K. V os, and T. Terriberry, “Definition of the Opus audio codec,” IETF , September, 2012

    work page 2012

  3. [3]

    Information technology—generic coding of moving pic- tures and associated audio information—-Part 7: Advanced audio coding (AAC),

    I. Standard, “Information technology—generic coding of moving pic- tures and associated audio information—-Part 7: Advanced audio coding (AAC),” ISO/IEC, pp. 13 818–7, 1997

    work page 1997

  4. [4]

    Joint source and channel coding,

    M. Fresia, F. Perez-Cruz, H. V . Poor, and S. Verdu, “Joint source and channel coding,” IEEE Signal Processing Magazine , vol. 27, no. 6, pp. 104–113, 2010

    work page 2010

  5. [5]

    Semantic communication systems for speech transmission,

    Z. Weng and Z. Qin, “Semantic communication systems for speech transmission,” IEEE Journal on Selected Areas in Communications , vol. 39, no. 8, pp. 2434–2444, 2021. 14

    work page 2021

  6. [6]

    Semantic-preserved communication system for highly efficient speech transmission,

    T. Han, Q. Yang, Z. Shi, S. He, and Z. Zhang, “Semantic-preserved communication system for highly efficient speech transmission,” IEEE Journal on Selected Areas in Communications , vol. 41, no. 1, pp. 245– 259, 2022

    2022

  7. [7]

    Nonlinear transform source-channel coding for semantic communications,

    J. Dai, S. Wang, K. Tan, et al. , “Nonlinear transform source-channel coding for semantic communications,” IEEE Journal on Selected Areas in Communications, vol. 40, no. 8, pp. 2300–2316, 2022

    work page 2022

  8. [8]

    Wireless deep speech semantic transmission,

    Z. Xiao, S. Yao, J. Dai, S. Wang, K. Niu, and P. Zhang, “Wireless deep speech semantic transmission,” in ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    work page 2023

Show all 48 references
  1. [9]

    Low-bitrate redundancy coding of speech using a rate-distortion-optimized variational autoencoder,

    J.-M. Valin, J. B ¨uthe, and A. Mustafa, “Low-bitrate redundancy coding of speech using a rate-distortion-optimized variational autoencoder,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

    2023

  2. [10]

    Effects of packet losses in waveform coded speech and improvements due to an odd-even sample-interpolation procedure,

    N. Jayant and S. Christensen, “Effects of packet losses in waveform coded speech and improvements due to an odd-even sample-interpolation procedure,” IEEE Transactions on Communications , vol. 29, no. 2, pp. 101–109, 1981

    1981

  3. [11]

    Adaptive Multi-Rate - Wideband (AMR-WB) speech codec; Er- ror concealment of erroneous or lost frames,

    3GPP, “Adaptive Multi-Rate - Wideband (AMR-WB) speech codec; Er- ror concealment of erroneous or lost frames,” 3rd Generation Partnership Project (3GPP), TS 26.191, 2022, version 17.0.0

    2022

  4. [12]

    A context en- coder for audio inpainting,

    A. Marafioti, N. Perraudin, N. Holighaus, and P. Majdak, “A context en- coder for audio inpainting,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 27, no. 12, pp. 2362–2372, 2019

    2019

  5. [13]

    End-to-end multi- loss training for low delay packet loss concealment,

    N. Li, X. Zheng, C. Zhang, L. Guo, and B. Yu, “End-to-end multi- loss training for low delay packet loss concealment,” in Proceedings of Interspeech 2022, 2022, pp. 585–589

    2022

  6. [14]

    Towards Error-Resilient Neural Speech Coding,

    H. Xue, X. Peng, X. Jiang, and Y . Lu, “Towards Error-Resilient Neural Speech Coding,” in Proceedings of Interspeech 2022 , 2022, pp. 4217– 4221

    2022

  7. [15]

    A Real-Time Wideband Neural V ocoder at 1.6kb/s Using LPCNet,

    J.-M. Valin and J. Skoglund, “A Real-Time Wideband Neural V ocoder at 1.6kb/s Using LPCNet,” in Proceedings of Interspeech 2019 , 2019, pp. 3406–3410

    2019

  8. [16]

    Wavenet based low rate speech coding,

    W. B. Kleijn, F. S. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters, “Wavenet based low rate speech coding,” in 2018 International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 676–680

    2018

  9. [17]

    Efficient and scal- able neural residual waveform coding with collaborative quantization,

    K. Zhen, M. S. Lee, J. Sung, S. Beack, and M. Kim, “Efficient and scal- able neural residual waveform coding with collaborative quantization,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 361–365

    2020

  10. [18]

    Variational speech waveform com- pression to catalyze semantic communications,

    S. Yao, Z. Xiao, S. Wang, et al. , “Variational speech waveform com- pression to catalyze semantic communications,” in 2023 IEEE Wireless Communications and Networking Conference (WCNC) . IEEE, 2023, pp. 1–6

    2023

  11. [19]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transac- tions on Audio, Speech and Language Processing , vol. 30, pp. 495–507, 2021

    2021

  12. [20]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research ,

  13. [21]

    Language modeling is compression,

    G. Del ´etang, A. Ruoss, et al., “Language modeling is compression,” in Proceedings of Twelfth International Conference on Learning Represen- tations, 2024

    2024

  14. [22]

    Deep generative modeling reshapes compression and transmission: From efficiency to resiliency,

    J. Dai, X. Qin, S. Wang, L. Xu, K. Niu, and P. Zhang, “Deep generative modeling reshapes compression and transmission: From efficiency to resiliency,” IEEE Wireless Communications , vol. 31, no. 4, pp. 48–56, 2024

    2024

  15. [23]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2019

  16. [24]

    Latent-domain predictive neural speech coding,

    X. Jiang, X. Peng, H. Xue, Y . Zhang, and Y . Lu, “Latent-domain predictive neural speech coding,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 41, pp. 2111–2123, 2023

    2023

  17. [25]

    Joint coding-modulation for digital semantic communications via variational autoencoder,

    Y . Bo, Y . Duan, S. Shao, and M. Tao, “Joint coding-modulation for digital semantic communications via variational autoencoder,” IEEE Transactions on Communications, vol. 72, no. 9, pp. 5626–5640, 2024

    2024

  18. [26]

    Deepjscc- q: Constellation constrained deep joint source-channel coding,

    T.-Y . Tung, D. B. Kurka, M. Jankowski, and D. G ¨und¨uz, “Deepjscc- q: Constellation constrained deep joint source-channel coding,” IEEE Journal on Selected Areas in Information Theory , vol. 3, no. 4, pp. 720–731, 2022

    2022

  19. [27]

    Robust semantic transmission of images with generative adversarial networks,

    Q. He, H. Yuan, D. Feng, B. Che, Z. Chen, and X.-G. Xia, “Robust semantic transmission of images with generative adversarial networks,” in GLOBECOM 2022-2022 IEEE Global Communications Conference . IEEE, 2022, pp. 3953–3958

    2022

  20. [28]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/ language-unsupervised/language understanding paper.pdf

    2018

  21. [29]

    Soundstorm: Efficient parallel audio generation,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio generation,” arXiv preprint arXiv:2305.09636 , 2023

    2023 arXiv

  22. [30]

    AudioGen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “AudioGen: Textually guided audio generation,” in Proceedings of Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    MAGVIT: Masked generative video transformer,

    L. Yu, Y . Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y . Hao, I. Essa, and L. Jiang, “MAGVIT: Masked generative video transformer,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 10 459– 10 469

    2023

  24. [32]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS’17. Red Hook, NY , USA: Curran Associates Inc., 2017, pp. 6000––6010

    2017

  25. [33]

    Neural discrete representation learning,

    A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Proceedings of the 31st International Confer- ence on Neural Information Processing Systems , 2017, pp. 6309–6318

    2017

  26. [34]

    RFC 7587: RTP payload format for the opus speech and audio codec,

    J. Spittka, K. V os, and J. Valin, “RFC 7587: RTP payload format for the opus speech and audio codec,” USA, 2015

    2015

  27. [35]

    Estimating or propagating gradients through stochastic neurons for conditional computation,

    Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013

    2013 arXiv

  28. [36]

    Melgan: Generative adversarial networks for conditional waveform synthesis,

    K. Kumar, R. Kumar, T. De Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. De Brebisson, Y . Bengio, and A. C. Courville, “Melgan: Generative adversarial networks for conditional waveform synthesis,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  29. [37]

    Librispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  30. [38]

    The mtg-jamendo dataset for automatic music tagging,

    D. Bogdanov, M. Won, et al. , “The mtg-jamendo dataset for automatic music tagging,” in Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019. [Online]. Available: http://hdl.handle.net/10230/42015

    2019

  31. [39]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  32. [40]

    An analysis of packet loss models for dis- tributed speech recognition,

    M. Ben and J. Alastair, “An analysis of packet loss models for dis- tributed speech recognition,” in 8th International Conference on Spoken Language Processing (Interspeech 2004 - ICSLP) , Korea, 2004

    2004

  33. [41]

    Tasnet: time-domain audio separation network for real-time, single-channel speech separation,

    Y . Luo and N. Mesgarani, “Tasnet: time-domain audio separation network for real-time, single-channel speech separation,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 696–700

    2018

  34. [42]

    SDR– half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR– half-baked or well done?” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 626–630

    2019

  35. [43]

    Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band tele- phone networks and speech codecs,

    ITU-T, “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band tele- phone networks and speech codecs,” International Telecommunication Union, 2001

    2001

  36. [44]

    ViSQOL v3: An open source production ready objective speech and audio metric,

    M. Chinen, F. S. Lim, J. Skoglund, et al. , “ViSQOL v3: An open source production ready objective speech and audio metric,” in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2020, pp. 1–6

    2020

  37. [45]

    PLCMOS – a data-driven non-intrusive metric for the evaluation of packet loss concealment algorithms,

    L. Diener, M. Purin, S. Sootla, et al. , “PLCMOS – a data-driven non-intrusive metric for the evaluation of packet loss concealment algorithms,” in Proceedins of Interspeech 2023 , 2023, pp. 2533–2537

    2023

  38. [46]

    Method for the subjective assessment of intermediate quality level of audio systems,

    B. Series, “Method for the subjective assessment of intermediate quality level of audio systems,” International Telecommunication Union Radio- communication Assembly, 2014

    2014

  39. [47]

    Lightweight deep learning for resource-constrained environments: A survey,

    H.-I. Liu, M. Galindo, H. Xie, L.-K. Wong, H.-H. Shuai, Y .-H. Li, and W.-H. Cheng, “Lightweight deep learning for resource-constrained environments: A survey,” ACM Computing Surveys, vol. 56, no. 10, pp. 1–42, 2024. [Online]. Available: https://doi.org/10.1145/3657282

    2024 doi

  40. [2023]

    Available: https://openreview.net/forum?id=ivCd8z8zR2

    [Online]. Available: https://openreview.net/forum?id=ivCd8z8zR2

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.