Softmax temperature controls the rank of learned representations: high temperature induces rank-deficit bias, compresses features, hurts OOD generalization, and improves OOD detection.
What Variables Affect Out-of-Distribution Generalization in Pretrained Models?
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Embeddings produced by pre-trained deep neural networks (DNNs) are widely used; however, their efficacy for downstream tasks can vary widely. We study the factors influencing transferability and out-of-distribution (OOD) generalization of pre-trained DNN embeddings through the lens of the tunnel effect hypothesis, which is closely related to intermediate neural collapse. This hypothesis suggests that deeper DNN layers compress representations and hinder OOD generalization. Contrary to earlier work, our experiments show this is not a universal phenomenon. We comprehensively investigate the impact of DNN architecture, training data, image resolution, and augmentations on transferability. We identify that training with high-resolution datasets containing many classes greatly reduces representation compression and improves transferability. Our results emphasize the danger of generalizing findings from toy datasets to broader contexts.
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization
Softmax temperature controls the rank of learned representations: high temperature induces rank-deficit bias, compresses features, hurts OOD generalization, and improves OOD detection.