Models trained against adversarial attacks first learn class-shared features, then forget them as robust overfitting sets in, and preserving these features explains why soft-label training helps.
On the Role of Discrete Tokenization in Visual Representation Learning
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In the realm of self-supervised learning (SSL), masked image modeling (MIM) has gained popularity alongside contrastive learning methods. MIM involves reconstructing masked regions of input images using their unmasked portions. A notable subset of MIM methodologies employs discrete tokens as the reconstruction target, but the theoretical underpinnings of this choice remain underexplored. In this paper, we explore the role of these discrete tokens, aiming to unravel their benefits and limitations. Building upon the connection between MIM and contrastive learning, we provide a comprehensive theoretical understanding on how discrete tokenization affects the model's generalization capabilities. Furthermore, we propose a novel metric named TCAS, which is specifically designed to assess the effectiveness of discrete tokens within the MIM framework. Inspired by this metric, we contribute an innovative tokenizer design and propose a corresponding MIM method named ClusterMIM. It demonstrates superior performance on a variety of benchmark datasets and ViT backbones. Code is available at https://github.com/PKU-ML/ClusterMIM.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Identifying and Understanding Cross-Class Features in Adversarial Training
Models trained against adversarial attacks first learn class-shared features, then forget them as robust overfitting sets in, and preserving these features explains why soft-label training helps.