REVIEW 36 cited by
SoundStorm: Efficient Parallel Audio Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as input the semantic tokens of AudioLM, and relies on bidirectional attention and confidence-based parallel decoding to generate the tokens of a neural audio codec. Compared to the autoregressive generation approach of AudioLM, our model produces audio of the same quality and with higher consistency in voice and acoustic conditions, while being two orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4. We demonstrate the ability of our model to scale audio generation to longer sequences by synthesizing high-quality, natural dialogue segments, given a transcript annotated with speaker turns and a short prompt with the speakers' voices.
Forward citations
Cited by 36 Pith papers
-
SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
SemBridge supervises autoregressive states with discrete semantic tokens during training, improving content fidelity of continuous-latent speech generation without changing inference.
-
ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
A speech language model trained only on audiobooks with explicit word-level prosody tokens displays emerging abilities in prosody-controlled generation, emphasis and emotion understanding, and prosodic consistency acr...
-
SoundSpring: Loss-Resilient Audio Transceiver with Dual-Functional Masked Language Modeling
A single masked language model can both entropy-code audio tokens for efficient transmission and predict missing tokens for packet loss concealment, outperforming traditional codecs in tests.
-
Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
ResGen predicts cumulative vector embeddings of masked RVQ tokens, decoupling generative sampling cost from token depth and improving FID and TTS metrics over autoregressive baselines.
-
Scaling Transformers for Low-Bitrate High-Quality Speech Coding
A scaled transformer codec with FSQ reaches state-of-the-art speech reconstruction at 400-700 bps, outperforming CNN/RVQ baselines.
-
DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
A LoRA-plus-convolution adaptation converts an AR TTS backbone into a confidence-ordered discrete diffusion model that improves WER and speed on limited data.
-
Luna-TTS Family Technical Report
Luna-TTS shows that a 0.6B masked-diffusion text-to-speech model pretrained on 1M hours can match or beat autoregressive systems on standard benchmarks while decoding in parallel or in 1.28-second streaming blocks.
-
A Unifying Perspective on Audio Generative Modeling: Latent Representations and Modeling Strategies
The deciding factors for modeling an audio latent are dependency horizon and conditional ambiguity, viewed jointly with the representation, not whether the latent is discrete or continuous.
-
StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis
A mobile-oriented 83M-parameter masked transformer with sparse phone-anchored temporal embeddings achieves RTF 0.08 and lower WER than MaskGCT/F5-TTS on Seed-TTS test sets.
-
Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models
Using audio-aware LLMs to judge event presence and temporal order as DPO rewards improves multi-event text-to-audio instruction following.
-
Fr\'echet Distance Loss on Speech Representations for Text-to-Speech Synthesis
Four-step SR-FD fine-tuning cuts VoxCPM2's Seed-TTS English WER from 2.23% to 1.41%, beating the ten-step baseline at 1.74% with no inference-time cost.
-
DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
Block discrete diffusion over X-Codec2 tokens yields competitive zero-shot TTS with 0.6B parameters, 20K training hours, and a 0.15 real-time factor.
-
LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation
LaDA-Band generates complete vocal-aligned instrumental accompaniments with discrete masked diffusion, claiming simultaneous gains in audio fidelity, long-range coherence, and orchestration quality.
-
SpectroStream: A Versatile Neural Codec for General Audio
SpectroStream, a 2D time-frequency neural codec, reconstructs 48 kHz stereo music at 4-16 kbps with better ViSQOL and subjective quality than DAC.
-
Scaling Self-Supervised Representation Learning for Symbolic Piano Performance
Self-supervised pretraining on 60,000 hours of symbolic piano music produces a generative model and contrastive embeddings that beat leading baselines on continuation quality and several MIR classification benchmarks.
-
DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding
DiffSoundStream uses a latent diffusion decoder conditioned on WavLM semantic tokens and coarse SoundStream acoustic tokens to match 100-token-per-second quality at 50 tokens per second.
-
LinearVC: Linear transformations of self-supervised features through the lens of voice conversion
A linear projection of WavLM features, including a rank-100 SVD factorization, performs voice conversion on par with much larger neural systems.
-
Learning to Upsample and Upmix Audio in the Latent Domain
Lightweight networks trained only on autoencoder latent codes can do bandwidth extension and mono-to-stereo upmixing at a fraction of the FLOPS of raw-audio models, but match those models only when the baselines are a...
-
Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation
Compressed-to-fine language modeling improves speech token prediction by retaining prompt and local tokens while compressing long-range token spans into compact summaries.
-
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation
VoiceStar uses a progress-based rotary position embedding and mixed prompt training to give zero-shot voice cloning precise duration control and much longer output than training clips.
-
A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model
A hierarchical speech enhancement pipeline that estimates semantic tokens first and acoustic tokens second, via a factorized codec and diffusion, improves DNSMOS and downstream TTS speaker similarity in noisy far-fiel...
-
Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment
Preference alignment on the new INTP dataset improves intelligibility and quality of zero-shot TTS across diverse domains, with weak-to-strong generalization shown on CosyVoice 2 and Ints.
-
SepALM: Audio Language Models Are Error Correctors for Robust Speech Separation
A four-stage pipeline (separate, correct in text, re-synthesize, align) improves speech separation quality and out-of-domain noise robustness.
-
GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling
GenSE enhances speech by first denoising semantic tokens with a language model and then generating acoustic tokens from a single-quantizer codec, reporting higher DNSMOS, speaker similarity, and lower WER than prior systems.
-
Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
Continuous autoregressive text-to-speech with a Gaussian-mixture codec matches or beats a discrete-codec VALL-E baseline with a fraction of the language model parameters.
-
CodecFake+: Codec-Based Resynthesized Data as a Proxy for Detecting CodecFake Speech
A new large-scale dataset and codec taxonomy show that codec re-synthesized speech, especially balanced by decoder type, trains detectors that catch codec-based deepfake speech better than traditional anti-spoofing training.
-
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters
SOLAMI is an end-to-end social vision-language-action model that takes a user's speech and body motion as input and generates a 3D character's spoken and gestural responses in one pass.
-
Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders
A post-hoc Re-Bottleneck network trained only in latent space can impose ordering, semantic alignment, or equivariance on pre-trained audio autoencoder latents with little extra compute.
-
Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
R-VC performs zero-shot voice conversion in two sampling steps while transferring the target speaker's rhythm, matching or exceeding prior systems in naturalness and intelligibility.
-
FUSE: Universal Speech Enhancement using Multi-Stage Fusion of Sparse Compression and Token Generation Models for the URGENT 2025 Challenge
A three-stage fusion of a sparse compression separator, an SSL-conditioned codec-token generator, and a fusion network achieves third place in URGENT 2025 and improves perceptual metrics with a modest fidelity trade-off.
-
Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages
Fine-tuning the English F5-TTS model on small Indian-language datasets yields a near-human-quality polyglot TTS (IN-F5) with voice cloning, code-mixing, and zero-resource synthesis for Bhojpuri and Tulu.
-
Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
A masked generative model pre-trained on unlabeled speech then fine-tuned per task matches or beats task-specific systems across TTS, voice conversion, speaker extraction, enhancement, and lip-to-speech.
-
ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training
ZSVC uses a speech codec and a latent diffusion model with a style prompt, plus an information bottleneck and adversarial training, to convert speaking style while preserving speaker identity in zero-shot settings.
-
Overview of the Amphion Toolkit (v0.2)
Amphion v0.2 is an open-source toolkit for audio, music, and speech generation, adding a 101K-hour multilingual dataset, processing pipelines, and pretrained models.
-
CycleFlow: Leveraging Cycle Consistency in Flow Matching for Speaker Style Adaptation
CycleFlow applies cycle-consistency training and a two-stage flow matching decoder to non-parallel voice conversion, but the cycle loss as written is not a cycle and the similarity metric is the same encoder used for ...
-
Watermarking across Modalities for Content Tracing and Generative AI
A thesis showing that invisible watermarks can be embedded across images, audio, text, and model weights, with statistical tests for tracing AI-generated content.
Discussion (0). Continue with ORCID to comment.