Lip Forcing distills a 14B bidirectional video diffusion teacher into autoregressive students that achieve real-time lip synchronization at 31 FPS using two denoising steps without CFG.
hub
URL http://proceedings.mlr.press/ v37/allamanis15.html
26 Pith papers cite this work. Polarity classification is still indexing.
26
Pith papers citing it
abstract
We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.
hub tools
citation-role summary
other 1
citation-polarity summary
roles
other 1polarities
unclear 1representative citing papers
TTS-generated poisoned audio implants backdoors in SER models achieving high attack success at low poisoning ratios while preserving clean performance.
Moshi is the first real-time full-duplex spoken large language model that casts dialogue as speech-to-speech generation using parallel audio streams and an inner monologue of time-aligned text tokens.
MADB is a 9,999-track music aesthetics benchmark with multi-dimensional professional annotations revealing that current pretrained audio models capture only partial aesthetic information.
An open-source benchmark for speech-to-speech models shows that current systems produce intelligible audio but diverge from human conversational behavior in latency, dialect consistency, emotional entrainment, and prosody.
An end-to-end SLU architecture with frozen SSL acoustic encoder, LSTM classification head, and cross-modal distillation achieves 93% accuracy on simple commands and 82% on spontaneous speech at 7 ms latency on the new VoiceStick corpus, outperforming cascade baselines.
PiDA generates phonetically similar corruptions for fine-tuning NMT on FLEURS Vietnamese-English, improving translation of ASR errors by up to +2.04 BLEU while slightly boosting clean performance.
Voxtral Realtime is an end-to-end trained streaming ASR model that achieves Whisper-level transcription quality at 480ms delay after scaling pretraining across 13 languages.
Step-Audio 2 integrates a latent audio encoder, reasoning-centric reinforcement learning, and discrete audio token generation into language modeling to deliver state-of-the-art performance on audio understanding and conversational benchmarks.
Atlas reaches over 42% accuracy on Natural Questions with only 64 examples, outperforming a 540B-parameter model by 3% with 50x fewer parameters.
Contrastive learning trains unsupervised dense retrievers that beat BM25 on most BEIR datasets and support cross-lingual retrieval across scripts.
Codex achieves 28.8% pass@1 on HumanEval, rising to 70.2% with 100 samples per problem via repeated sampling.
wav2VOT shows wav2vec2 can estimate voice onset time and related stop consonant features with accuracy comparable to existing tools on unseen data and higher accuracy after fine-tuning.
A wav2vec2 CTC recognizer with alignment-based error typing screens for sibilant substitutions in Polish children, reporting 88.7% sequence match and F1=0.67 on 559 utterances from 10 unseen children.
CoughPhase-CLR uses cough physiological phases to build contrastive positive pairs, outperforming random cropping on downstream tasks including COVID-19 detection and COPD classification.
F3-Tokenizer adapts audio autoencoder latents with noise-regularized bottleneck (channel normalization and stochastic perturbation) and a representation encoder (RQ-MTP plus frozen-LLM supervision) to support both high-dimensional understanding representations and normalized continuous generation ta
CAFNet performs joint ternary classification and temporal boundary regression for half-truth audio deepfakes via cross-attentive fusion of MFCC, LFCC, and Chroma-STFT features, reporting 92.71% accuracy and 0.075s MAE on MLADDC T2+T3.
A hybrid confidence-aware ASR training framework with learnable weights reduces Telugu medical WER from 24.3% to 15.8% and Kannada from 31.7% to 25.4%, outperforming standard fine-tuning.
MMTalker maps speech to 3D face motion via multi-resolution UV sampling and residual-GCN plus dual cross-attention multimodal fusion, reporting better lip and eye sync than prior methods.
Machine interpreting should shift from fidelity metrics to three design priorities—agency, grounding, and experience—drawn from interpreting studies to close the usability gap with human-mediated communication.
Fine-tuned Wav2Vec2 and HuBERT models recognize click consonants more accurately than non-clicks in G|ui and West !Xoon data.
SPARCLE builds speaker-aware grapheme representations by contrastively aligning characters with Wav2Vec2 acoustic embeddings conditioned on speaker identity, replacing G2P for TTS and halving WER in low-resource cases.
Domain-adapted ECG foundation models with self-supervised pretraining and selective fine-tuning reach macro-AUROC 0.8509 for multi-label structural heart disease detection on the EchoNext benchmark.
SpAArSIST sparsifies AASIST by swapping learned pooling for explicit magnitude-based scoring and mean aggregation, cutting compute 20.7% and improving In-the-Wild EER to 2.82%.
citing papers explorer
-
Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
Lip Forcing distills a 14B bidirectional video diffusion teacher into autoregressive students that achieve real-time lip synchronization at 31 FPS using two denoising steps without CFG.
-
Backdoor Attacks on Speech Emotion Recognition via TTS-Generated Poisoning
TTS-generated poisoned audio implants backdoors in SER models achieving high attack success at low poisoning ratios while preserving clean performance.
-
Moshi: a speech-text foundation model for real-time dialogue
Moshi is the first real-time full-duplex spoken large language model that casts dialogue as speech-to-speech generation using parallel audio streams and an inner monologue of time-aligned text tokens.
-
MADB: A Large-Scale Music Aesthetics Dataset with Professional and Multi-Dimensional Annotations
MADB is a 9,999-track music aesthetics benchmark with multi-dimensional professional annotations revealing that current pretrained audio models capture only partial aesthetic information.
-
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
An open-source benchmark for speech-to-speech models shows that current systems produce intelligible audio but diverge from human conversational behavior in latency, dialect consistency, emotional entrainment, and prosody.
-
End-to-End Voice Intent Recognition for Spontaneous Human-Drone Interaction with Naive Users
An end-to-end SLU architecture with frozen SSL acoustic encoder, LSTM classification head, and cross-modal distillation achieves 93% accuracy on simple commands and 82% on spontaneous speech at 7 ms latency on the new VoiceStick corpus, outperforming cascade baselines.
-
PiDA: Phonetically-Informed Data Augmentation for Robust Vietnamese Speech Translation
PiDA generates phonetically similar corruptions for fine-tuning NMT on FLEURS Vietnamese-English, improving translation of ASR errors by up to +2.04 BLEU while slightly boosting clean performance.
-
Voxtral Realtime
Voxtral Realtime is an end-to-end trained streaming ASR model that achieves Whisper-level transcription quality at 480ms delay after scaling pretraining across 13 languages.
-
Step-Audio 2 Technical Report
Step-Audio 2 integrates a latent audio encoder, reasoning-centric reinforcement learning, and discrete audio token generation into language modeling to deliver state-of-the-art performance on audio understanding and conversational benchmarks.
-
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Atlas reaches over 42% accuracy on Natural Questions with only 64 examples, outperforming a 540B-parameter model by 3% with 50x fewer parameters.
-
Unsupervised Dense Information Retrieval with Contrastive Learning
Contrastive learning trains unsupervised dense retrievers that beat BM25 on most BEIR datasets and support cross-lingual retrieval across scripts.
-
Evaluating Large Language Models Trained on Code
Codex achieves 28.8% pass@1 on HumanEval, rising to 70.2% with 100 samples per problem via repeated sampling.
-
wav2VOT: Automatic estimation of voice onset time, closure duration, and burst realisation with wav2vec2
wav2VOT shows wav2vec2 can estimate voice onset time and related stop consonant features with accuracy comparable to existing tools on unseen data and higher accuracy after fine-tuning.
-
Phoneme-Level Mispronunciation Screening in Polish-Speaking Children with an Explainable Assistant
A wav2vec2 CTC recognizer with alignment-based error typing screens for sibilant substitutions in Polish children, reporting 88.7% sequence match and F1=0.67 on 559 utterances from 10 unseen children.
-
CoughPhase-CLR: Designing an acoustics-informed foundation model for coughing sound classification
CoughPhase-CLR uses cough physiological phases to build contrastive positive pairs, outperforming random cropping on downstream tasks including COVID-19 detection and COPD classification.
-
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
F3-Tokenizer adapts audio autoencoder latents with noise-regularized bottleneck (channel normalization and stochastic perturbation) and a representation encoder (RQ-MTP plus frozen-LLM supervision) to support both high-dimensional understanding representations and normalized continuous generation ta
-
Audio Deepfake Detection with Half-Truth Localisation Using Cross-Attentive Feature Fusion
CAFNet performs joint ternary classification and temporal boundary regression for half-truth audio deepfakes via cross-attentive fusion of MFCC, LFCC, and Chroma-STFT features, reporting 92.71% accuracy and 0.075s MAE on MLADDC T2+T3.
-
Enhancing ASR Performance in the Medical Domain for Dravidian Languages
A hybrid confidence-aware ASR training framework with learnable weights reduces Telugu medical WER from 24.3% to 15.8% and Kannada from 31.7% to 25.4%, outperforming standard fine-tuning.
-
MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion
MMTalker maps speech to 3D face motion via multi-resolution UV sampling and residual-GCN plus dual cross-attention multimodal fusion, reporting better lip and eye sync than prior methods.
-
Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Design
Machine interpreting should shift from fidelity metrics to three design priorities—agency, grounding, and experience—drawn from interpreting studies to close the usability gap with human-mediated communication.
-
Pretrained self-supervised speech models can recognize unseen consonants
Fine-tuned Wav2Vec2 and HuBERT models recognize click consonants more accurately than non-clicks in G|ui and West !Xoon data.
-
SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings
SPARCLE builds speaker-aware grapheme representations by contrastively aligning characters with Wav2Vec2 acoustic embeddings conditioned on speaker identity, replacing G2P for TTS and halving WER in low-resource cases.
-
Domain-Adapted Fine-Tuning of ECG Foundation Models for Multi-Label Structural Heart Disease Screening
Domain-adapted ECG foundation models with self-supervised pretraining and selective fine-tuning reach macro-AUROC 0.8509 for multi-label structural heart disease detection on the EchoNext benchmark.
-
SpAArSIST: Sparsified AASIST for Efficient and Reliable Anti-Spoofing
SpAArSIST sparsifies AASIST by swapping learned pooling for explicit magnitude-based scoring and mean aggregation, cutting compute 20.7% and improving In-the-Wild EER to 2.82%.
-
Spatial Speech Perception Systems: A Survey of Sound Source Localization, Directional Enhancement, and Speech Recognition
A survey of spatial speech perception systems covering sound source localization, directional enhancement, and automatic speech recognition methods and their integration.
-
Exploitation of Hidden Context in Dynamic Movement Forecasting: A Neural Network Journey from Recurrent to Graph Neural Networks and General Purpose Transformers
Empirical comparison of LSTM, GNN, and Transformer architectures for NBA trajectory forecasting finds hybrid LSTM with contextual information yields lowest FDE of 1.51m over horizons up to 2s.