REVIEW 12 cited by
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
read the original abstract
Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.
Forward citations
Cited by 12 Pith papers
-
Interleaved Speech Language Models Latently Work In Text
Interleaved SLMs implicitly transcribe spoken words to text tokens in middle layers (top candidate for 77% of data) before predicting in text space and returning to speech.
-
Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild
A new dataset and benchmark maps movie clips to distributions of audience emotional reactions derived from YouTube comments, showing that finetuned vision-language models can predict these distributions from video alone.
-
Syntactic Belief Update as the Driver of Garden Path Processing Difficulty
Syntactic belief update via generalized Rényi divergence on syntactic trees predicts garden path reading times better than lexical surprisal.
-
End-to-End Voice Intent Recognition for Spontaneous Human-Drone Interaction with Naive Users
An end-to-end SLU architecture with frozen SSL acoustic encoder, LSTM classification head, and cross-modal distillation achieves 93% accuracy on simple commands and 82% on spontaneous speech at 7 ms latency on the new...
-
Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG
Fine-tuning language encoding models on fMRI responses improves prediction performance for ECoG brain signals in frequency bands beyond fMRI resolution.
-
Meow-Omni 1: A Multimodal Large Language Model for Feline Ethology
Meow-Omni 1 is a quad-modal MLLM that fuses video, audio, physiological time-series, and text to achieve 71.16% accuracy on feline intent recognition in the new MeowBench benchmark.
-
Neural networks for Text-to-Speech evaluation
NeuralSBS reaches 73.7% accuracy on side-by-side TTS comparisons and enhanced MOS models reach RMSE 0.40, beating the human inter-rater baseline of 0.62.
-
Step-Audio 2 Technical Report
Step-Audio 2 integrates a latent audio encoder, reasoning-centric reinforcement learning, and discrete audio token generation into language modeling to deliver state-of-the-art performance on audio understanding and c...
-
Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal
A framework using native-only trained discrete token surprisal and DTW alignment features improves pronunciation assessment PCC to 0.66 on SpeechOcean762, approaching supervised performance.
-
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
F3-Tokenizer adapts audio autoencoder latents with noise-regularized bottleneck (channel normalization and stochastic perturbation) and a representation encoder (RQ-MTP plus frozen-LLM supervision) to support both hig...
-
Pretrained self-supervised speech models can recognize unseen consonants
Fine-tuned Wav2Vec2 and HuBERT models recognize click consonants more accurately than non-clicks in G|ui and West !Xoon data.
-
A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition
A dual-modality model combining DINOv2 visual features with Wav2Vec audio features achieves Macro-F1 of 0.5368 on the ABAW validation set for facial expression recognition.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.