Pith. sign in

REVIEW 4 major objections 6 minor 48 references

BRIDLE: Generalized Self-supervised Learning with Quantization

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BRIDLE, a self-supervised pretraining framework that replaces a single vector-quantization codebook with four hierarchical residual-quantization codebooks, consistently improves downstream classification accuracy across audio, image, and…

desk verdict The paper shows consistent gains from a 4-codebook quantizer over VQ, but the comparison confounds residual hierarchy with a 4x-longer target token sequence, so the central claim is not yet established. read the letter →

arxiv 2502.02118 v1 pith:KE7MAUHU submitted 2025-02-04 cs.LG cs.CV

Hoang M. Nguyen , Satya N. Shukla , Qiang Zhang , Hanchao Yu , Sreya D. Roy , Taipeng Tian , Lingjiong Zhu , Yuchen Liu This is my paper · ORCID
classification cs.LGcs.CV
keywords self-supervisedlearningresidualquantizationvectoraudioclassificationimagevideocodebookBEATs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BRIDLE is a self-supervised pretraining method that extends the BEATs bidirectional training recipe by replacing single-codebook vector quantization (VQ) with residual quantization (RQ): four smaller codebooks that hierarchically quantize the difference between a latent vector and the codes already chosen. The paper tries to establish that this swap, together with codebook-level tricks (k-means initialization, EMA updates, resetting unused codes), yields better downstream classifiers than VQ across three modalities. On AudioSet-2M fine-tuning, mAP rises from 47.64 (BEATs/VQ iteration 2) to 47.99 (BRIDLE/RQ with k-means); on ImageNet-1K, Top-1 accuracy rises from 80.26 to 81.10; on Kinetics-400, from 71.32 to 72.90. The authors argue that multiple residual stages capture finer latent structure and use codebooks more evenly, and they report state-of-the-art audio classification results. Why it matters: if correct, a simple quantizer change inside an existing bidirectional-encoding pipeline improves unlabeled pretraining for at least three data types without changing the encoder architecture.

What carries the argument

The load-bearing mechanism is residual quantization (RQ): given a latent vector $z_t$, stage $m$ picks the code vector in codebook $C_m$ closest to the current residual $e_m$, subtracts it, and passes $e_{m+1}$ to the next codebook; the quantized token is the sum of the $M$ selected code vectors. In BRIDLE this runs inside the BEATs-style interleaved self-distillation: the tokenizer (encoder plus RQ codebooks plus a tokenizer estimator) is trained to predict the main encoder's embeddings, and the main encoder plus decoder is trained to predict the tokenizer's codes on masked inputs. Supporting machinery includes EMA codebook updates, k-means code initialization, and resetting rarely used codes, all aimed at keeping codebooks used and stable. The comparison that carries the claim is VQ with one 1,024-code codebook versus RQ with four 256-code codebooks, matched on total code count.

What would settle it

Run a BRIDLE variant whose tokenizer, like VQ, assigns each latent to one code but whose decoder predicts a four-token sequence—for instance, by tokenizing the same latent with four independent 256-code codebooks selected in parallel rather than recursively; if that variant matches BRIDLE-RQ's numbers (AudioSet-2M FT mAP about 47.99, ImageNet-1K Top-1 about 81.10), the reported gains need not be attributed to residual quantization.

Watch

Extended reading notes

Core claim

The central claim, stated as the authors would state it, is that residual quantization is a better tokenizer for bidirectional self-distilled pretraining than vector quantization when total code capacity is held fixed. BRIDLE pretrains a ViT encoder to predict masked token sequences, while a tokenizer is trained in alternating phases to map the encoder's latents into codes; the tokenizer's mapping is what changes. Instead of one codebook of 1,024 codes, BRIDLE uses four codebooks of 256 codes each, selecting one code per stage against the running residual error and summing the selected codes into the final quantized representation. Under this design, each latent maps to four codes, enriching the target sequence the decoder must predict. Across AudioSet-2M, ImageNet-1K, and Kinetics-400, fine-tuning and especially linear probing improve relative to the VQ equivalent, and audio numbers are claimed to be state of the art. The paper's explanation is that hierarchical residual stages allow finer discretization of the latent space and near-full codebook utilization (roughly 100% Code Usage Rate) rather than the skewed usage typical of a single large codebook.

Load-bearing premise

The load-bearing premise is that matching the total number of code vectors (1,024 in one VQ codebook versus 4×256 in RQ) makes the comparison fair; but since RQ makes each latent produce four codes while VQ produces one, the four-times-larger prediction target, not the residual mechanism, could explain the gains.

Editorial extensions

If this is right

  • In audio, BRIDLE/RQ with k-means reaches 47.99 mAP on AudioSet-2M fine-tuning and 38.08 on AudioSet-20K, matching or beating the reproduced BEATs VQ results and giving the paper's claimed state-of-the-art audio classification.
  • On ImageNet-1K, RQ raises fine-tuning Top-1 from 80.26 (VQ) to 81.10 and linear-probing Top-1 from 53.21 to 56.30, so the representation is more linearly separable after the same pretraining budget.
  • On Kinetics-400, RQ raises Top-1 from 71.32 (VQ) to 72.90, extending the benefit to spatiotemporal data.
  • Across modalities, RQ codebooks achieve near-100% Code Usage Rate and higher Effective Code Usage than VQ, indicating more balanced tokenizer usage.
  • K-means codebook initialization consistently outperforms uniform initialization, and EMA updates outperform straight backpropagation, so codebook initialization and update rules are part of the reported gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because RQ maps each latent to four codes while VQ maps it to one, the compared variants differ in target-sequence richness, not only in quantizer structure; matching the number of predicted tokens (for instance, predicting four codes from a single VQ-style codebook) would tell whether the gain is really from residual refinement.
  • Editorial inference: the paper's joint-training experiments show VQ tolerates simultaneous encoder/tokenizer updates but RQ does not; an underexplored direction is a scheduled or multi-rate update scheme that gives RQ the same efficiency.
  • Editorial inference: the codebook practices isolated here (k-means init, EMA, reset) are quantizer-agnostic and could transfer to other discrete-token SSL pipelines, such as masked image modeling with VQ tokenizers, independent of BRIDLE's bidirectional objective.
  • Editorial inference: the AudioSet numbers are measured on a 2023 download with roughly 15% of clips missing; direct comparisons to other published audio SSL results would need the same data split, and the paper's state-of-the-art claim is relative to its own BEATs reproduction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BRIDLE, a self-supervised pretraining framework that replaces the vector quantization (VQ) tokenizer in the BEATs-style bidirectional encoder with residual quantization (RQ) using four hierarchical codebooks of 256 code vectors each. The encoder and tokenizer are trained in an interleaved, self-distilled manner, and the framework is applied to audio (AudioSet, ESC-50), image (ImageNet-1K), and video (Kinetics-400). The paper reports consistent downstream gains for RQ over a VQ baseline across all three modalities, together with codebook analyses (code usage rate, effective code usage) and an Appendix B proof of convergence for the exponential-moving-average codebook update. The central claim is that residual quantization, rather than a larger number of discrete targets, is responsible for the improvements, and the paper further claims state-of-the-art audio results.

Significance. If the improvement were convincingly attributable to residual quantization, the paper would be a useful engineering contribution: it shows that a simple tokenizer change can improve SSL representations across three modalities, documents practical codebook training techniques, and releases code and hyperparameters. The experiments span multiple benchmarks and include both fine-tuning and linear probing, and the codebook utilization analysis is informative. However, the central attribution is not yet established: the RQ-vs-VQ comparison varies the number of predicted tokens simultaneously with the residual hierarchy, no variance estimates are reported, and the audio state-of-the-art claim rests on a single reproduced baseline. The theoretical appendix is conditional and contains an algebraic error. These are fixable, but they require additional controlled experiments and revised claims.

major comments (4)
  1. [Section 4.2, Eq. (4), Tables 2-4] The RQ-versus-VQ comparison is confounded. Matching the total number of code vectors (1x1024 for VQ vs 4x256 for RQ) does not match the encoder's prediction task: under Eq. (4), the encoder loss sums over M codebooks, so at each masked position the RQ model predicts four token IDs (with a per-position label space of 256^4) while the VQ model predicts one token ID (from 1024). All headline gains (e.g., Table 2 AS-2M FT mAP 47.99 vs 47.64; Table 3 ImageNet FT Top-1 81.10 vs 80.26; Table 4 Kinetics-400 Top-1 72.90 vs 71.32) therefore change two factors at once: the residual hierarchy and the number/richness of discrete prediction targets. Since no ablation with a non-residual multi-codebook quantizer (e.g., four independent 256-code codebooks) is reported, the paper does not establish that the gains are due to residual quantization rather than to the larger target representation. Section 5's informal remarks about increasing the number of codes or using soft codes do not control for this, because they also change the assignment mechanism.
  2. [Tables 2-4] All reported results appear to be single runs with no variance estimates, confidence intervals, or multiple seeds. The improvements central to the paper are small (0.35 mAP in Table 2, 0.84 Top-1 in Table 3, 1.58 Top-1 in Table 4), and without repeated-run statistics the claim of "consistent improvements" is not statistically supported. I would like to see at least two or three seeds with mean and standard deviation for the headline comparisons, or a clear statement of computational constraints if that is infeasible.
  3. [Appendix B, Proposition 3; Section 3.3] The claimed convergence result for the EMA codebook update assumes that the quantizer assignments q_{j,t} and latent vectors z_{j,t} converge almost surely. Since q_{j,t} is determined by the codebooks whose convergence is the object of study, the proposition proves only that the EMA recursion is stable given convergent inputs; it does not prove that codebook learning converges. This is a circularity in the claimed "theoretical contribution" in Section 3.3. In addition, Lemma 2's bound B_N=S is algebraically inconsistent with Eq. (14), which contains a +epsilon term: the induction step yields at most S+epsilon, so the bound should be S+epsilon/(1-gamma) (or epsilon should be removed).
  4. [Section 4.5; Abstract and Conclusion] The claim of state-of-the-art audio results is not supported by the reported comparisons. Table 2 only compares BRIDLE with two reproductions of BEATs; there is no comparison to other recent audio SSL models (e.g., AudioMAE, SSAST, MAE-AST, or HTSAT) on the same data, and the AS-2M evaluation set is about 85% of the original dataset (Table 1), which makes the relation to published BEATs numbers unclear. The conclusion's "achieved state-of-the-art results on audio classification benchmarks" should be replaced by a claim limited to the baselines actually evaluated, or supported by broader comparisons.
minor comments (6)
  1. [Eq. (4)] The loss is written as a sum over all T frames, but training uses masked prediction (mask ratio 0.8 or 0.9 in Table 5); an indicator over masked positions is missing.
  2. [Table 2] The ESC-50 column header "FT mAP | Acc" is unclear, since ESC-50 is a single-label dataset; the paper should state whether mAP is computed and, if so, how.
  3. [Section 5] The sentence reporting performance degradation from larger codebooks and soft codes gives no numbers; a small quantitative table would make this claim checkable.
  4. [Section 4.1] The statement that the AudioSet data loss "may introduce variations in performance metrics" should be revisited in the comparisons: since BEATs was trained on the full set, both the reproduction and BRIDLE numbers should be interpreted with this mismatch in mind.
  5. [Appendix B] The notation uses q_{j,t} as a scalar even though each latent maps to M codes; the equations should carry the codebook index m (or explicitly state the convention for each m).
  6. [Table 5] The "FT Masking 2D, ratio 0.2" entry for image is not defined; please explain the masking pattern.

Circularity Check

1 steps flagged · score 3.0 of 10

Central claim is empirical and self-contained, but Appendix B's EMA convergence 'theorem' assumes the very convergence it purports to prove; the RQ-vs-VQ comparison also confounds residual hierarchy with 4x token prediction targets, though that confound is an experimental design issue rather than a circular derivation.

  1. self definitional [Appendix B, Proposition 3; advertised in Section 3.3]
    "Section 3.3: 'In Appendix B, we show the convergence of the EMA update for all potential codebooks, given minor assumptions on the convergence of latent vectors and sufficiently large number of iterations.' Appendix B, Proposition 3: 'Assume that for every j = 1, 2, . . . , S, qj,t and zj,t converge almost surely as t → ∞. Then, (Ni,t, mi,t, ci,t) converges almost surely to (Ni,∞, mi,∞, ci,∞) as t → ∞ for every i = 1, 2, . . . , Km.'"

    The paper presents Proposition 3 as a novel theoretical proof of EMA convergence for codebook training. But qj,t is the discrete code assignment selected by argmin over the very codebooks ci,t that the EMA update is supposed to make converge (Eq. (1)). Assuming qj,t converges almost surely is therefore assuming that the quantizer's output—the central object whose stabilization the analysis is meant to establish—has already converged. Once that assumption is granted, the proof merely solves the linear recurrences (14)-(15) for a fixed point; it provides no argument that EMA drives the codebook assignments to convergence from arbitrary initialization.

full rationale

The paper's main performance claim—that BRIDLE with residual quantization (RQ) improves over a vector-quantization (VQ) baseline in audio, image, and video—is supported by self-contained experiments against an external baseline (BEATs) and by internal RQ-versus-VQ comparisons within the same framework; no fitted parameter is relabeled as a prediction, and no load-bearing uniqueness argument or author self-citation chain is invoked. The principal circularity found is in Appendix B, where Proposition 3 assumes convergence of qj,t and zj,t in order to conclude convergence of the EMA updates for codebooks; since qj,t is determined by the codebooks being trained, this is a self-definitional side claim rather than a proof of EMA convergence. Separately, the RQ/VQ comparison is confounded: Eq. (4) sums the masked cross-entropy over M codebooks, so with M=4 for RQ versus M=1 for VQ, RQ predicts four token IDs per latent position instead of one, and the comparison therefore does not isolate the residual mechanism from a much richer discrete prediction target. That is an experimental confound affecting attribution of the gains, but it is not a circular derivation. Consistent with this, the score of 3 reflects one self-definitional theoretical step while the central empirical claim retains independent content.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities, particles, forces, or conserved quantities. The ledger is dominated by hand-chosen hyperparameters (codebook count and size, loss weights, EMA decay) and inherited framework assumptions from BEATs and RQ-VAE. The only mathematically novel-looking claim, the EMA convergence proof, relies on an assumption that effectively presupposes its conclusion.

free parameters (6)
  • Number of codebooks M = 4 for RQ = 4
    Hand-chosen; the paper reports degraded performance when increasing the number of codes or using soft codes (Section 5). The VQ/RQ comparison depends on this setting.
  • Codebook size Km = 256 for RQ = 256
    Hand-chosen to keep total code count equal to VQ's 1024; the paper notes that larger codebooks can degrade performance (Section 5).
  • Commitment loss weight beta = Not stated explicitly
    Hyperparameter in Eq. (5); not ablated in the paper.
  • Cosine loss weight lambda_cos = Not stated explicitly
    Weights the cosine similarity loss in Eq. (7); not ablated.
  • EMA decay rate gamma = 0.99
    Standard EMA decay; chosen, not derived. Convergence proof in Appendix B depends on it being in (0,1).
  • Code reset threshold T_u = 1
    Threshold for resetting unused code vectors; hand-set.
assumptions (3)
  • domain assumption Latent vectors z_{j,t} and code assignments q_{j,t} converge almost surely as t -> infinity
    This is the central premise of the EMA convergence proof in Appendix B (Proposition 3). It assumes the convergence that the proof is meant to establish, making the theoretical contribution circular.
  • domain assumption Bidirectional masked prediction with self-distillation (the BEATs framework) produces useful representations
    BRIDLE inherits the entire interleaved training and masked prediction setup from BEATs [CWW+23]; the paper does not re-derive or justify this assumption.
  • domain assumption Discrete token prediction is a useful SSL target for images and video, not just audio
    The generalization to ImageNet and Kinetics assumes that masked token prediction with a quantized tokenizer transfers to visual modalities, following AudioMAE and VideoMAE setups without independent validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BRIDLE: Generalized Self-supervised Learning with Quantization." pith.science (2026). https://pith.science/paper/KE7MAUHU

@misc{pith2026250202118,
  author       = {Pith},
  title        = {Pith review of: BRIDLE: Generalized Self-supervised Learning with Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KE7MAUHU}},
  note         = {Machine review of arXiv:2502.02118}
}
read the original abstract

Self-supervised learning has been a powerful approach for learning meaningful representations from unlabeled data across various domains, reducing the reliance on large labeled datasets. Inspired by BERT's success in capturing deep bidirectional contexts in natural language processing, similar frameworks have been adapted to other modalities such as audio, with models like BEATs extending the bidirectional training paradigm to audio signals using vector quantization (VQ). However, these frameworks face challenges, notably their dependence on a single codebook for quantization, which may not capture the complex, multifaceted nature of signals. In addition, inefficiencies in codebook utilization lead to underutilized code vectors. To address these limitations, we introduce BRIDLE (Bidirectional Residual Quantization Interleaved Discrete Learning Encoder), a self-supervised encoder pretraining framework that incorporates residual quantization (RQ) into the bidirectional training process, and is generalized for pretraining with audio, image, and video. Using multiple hierarchical codebooks, RQ enables fine-grained discretization in the latent space, enhancing representation quality. BRIDLE involves an interleaved training procedure between the encoder and tokenizer. We evaluate BRIDLE on audio understanding tasks using classification benchmarks, achieving state-of-the-art results, and demonstrate competitive performance on image classification and video classification tasks, showing consistent improvements over traditional VQ methods in downstream performance.

Figures

Figures reproduced from arXiv: 2502.02118 by the authors.

Figure 1
Figure 1. BRIDLE training framework, which contains an encoder training phase (left), and [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. VQ codebook embeddings norm distributions before and after training on AudioSet [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗
Figure 3
Figure 3. RQ codebook embeddings norm distributions before and after training on AudioSet [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 40 canonical work pages

  1. [1]

    K- SVD : An algorithm for designing overcomplete dictionaries for sparse representation

    Michal Aharon, Michael Elad, and Alfred Bruckstein. K- SVD : An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing , 54(11):4311--4322, 2006

    work page 2006

  2. [2]

    B E i T : BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. B E i T : BERT pre-training of image transformers. In International Conference on Learning Representations , 2022

    work page 2022

  3. [3]

    Audio LM : a language modeling approach to audio generation

    Zal \'a n Borsos, Rapha \"e l Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audio LM : a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 31:2523--2533, 2023

    work page 2023

  4. [4]

    vq-wav2vec: Self-supervised learning of discrete speech representations

    Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations (ICLR) , 2020

    work page 2020

  5. [5]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems , volume 33, pages 12449--12460, 2020

    work page 2020

  6. [6]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 1597--1607. PMLR, 2020

    work page 2020

  7. [7]

    Self-supervised learning with random-projection quantizer for speech recognition

    Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 3915--3924. PMLR, 2022

    work page 2022

  8. [8]

    BEATs : audio pre-training with acoustic tokenizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs : audio pre-training with acoustic tokenizers. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 5178--5193. PMLR, 2023

    work page 2023

Show all 48 references
  1. [9]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299--6308, 2017

    2017

  2. [10]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North A merican Chapter ...

    2019

  3. [11]

    Image N et: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Image N et: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248--255. IEEE, 2009

    2009

  4. [12]

    Jukebox: A generative model for music

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 , 2020

    2005 arXiv

  5. [13]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  6. [14]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2017

  7. [15]

    Vector Quantization and Signal Compression , volume 159

    Allen Gersho and Robert M Gray. Vector Quantization and Signal Compression , volume 159. Springer Science & Business Media, 2012

    2012

  8. [16]

    Optimized product quantization for approximate nearest neighbor search

    Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2946--2953, 2013

    2013

  9. [17]

    Gray and David L

    Robert M. Gray and David L. Neuhoff. Quantization. IEEE Transactions on Information Theory , 44(6):2325--2383, 1998

    1998

  10. [18]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch \'e , Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R\' e mi Munos, and Michal Valko. Bootstrap your own la...

    2020

  11. [19]

    Hu BERT : Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hu BERT : Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , ...

    2021

  12. [20]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000--16009, 2022

    2022

  13. [21]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729--9738, 2020

    2020

  14. [22]

    Masked autoencoders that listen

    Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In Advances in Neural Information Processing Systems , volume 35, pages 28708--28720, 2022

    2022

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770--778, 2016

    2016

  16. [24]

    Product quantization for nearest neighbor search

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(1):117--128, 2010

    2010

  17. [25]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017

    2017 arXiv

  18. [26]

    Image N et classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Image N et classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems , volume 25, 2012

    2012

  19. [27]

    High-fidelity audio compression with improved RVQGAN

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN . In Advances in Neural Information Processing Systems , volume 36, 2024

    2024

  20. [28]

    Audio G en: Textually guided audio generation

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D \'e fossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audio G en: Textually guided audio generation. In The Eleventh International Conference on Learning Representations , 2023

    2023

  21. [29]

    Robust training of vector quantized bottleneck models

    Adrian a \'n cucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alum \"a e, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN) ,...

    2020

  22. [30]

    Unsupervised representation learning by sorting sequences

    Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision , pages 667--676, 2017

    2017

  23. [31]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523--11532, 2022

    2022

  24. [32]

    Shuffle and learn: unsupervised learning using temporal order verification

    Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part I 14 , pages 527--544. ...

    2016

  25. [33]

    BEiT v2: Masked image modeling with vector-quantized visual tokenizers

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366 , 2022

    2022 arXiv

  26. [34]

    ESC : Dataset for environmental sound classification

    Karol J Piczak. ESC : Dataset for environmental sound classification. In Proceedings of the 23rd ACM International Conference on Multimedia , pages 1015--1018, 2015

    2015

  27. [35]

    Spatiotemporal contrastive video representation learning

    Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6964--6974, 2021

    2021

  28. [36]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Image N et large scale visual recognition challenge. International Journal of Computer Vision , ...

    2015

  29. [37]

    Generating diverse high-fidelity images with VQ-VAE-2

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2 . In Advances in Neural Information Processing Systems , volume 32, 2019

    2019

  30. [38]

    Contrastive learning of general-purpose audio representations

    Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3875--3879. IEEE, 2021

    2021

  31. [39]

    A mathematical theory of communication

    Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal , 27(3):379--423, 1948

    1948

  32. [40]

    Video BERT : A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Video BERT : A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7464--7473, 2019

    2019

  33. [41]

    Two-stream convolutional networks for action recognition in videos

    Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems , volume 27, 2014

    2014

  34. [42]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015

    2015

  35. [43]

    Learning spatiotemporal features with 3D convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3D convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 4489--4497, 2015

    2015

  36. [44]

    Video MAE : Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Video MAE : Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems , volume 35, pages 10078--10093, 2022

    2022

  37. [45]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  38. [46]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems , volume 30, 2017

    2017

  39. [47]

    Image as a foreign language: BEIT pretraining for vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEIT pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Confer...

    2023

  40. [48]

    Sound S tream: An end-to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound S tream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:495--507, 2021

    2021

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.