KCR-Transformer replaces standard transformer blocks with channel-pruned MLPs plus a truncated-nuclear-norm regularizer, reporting better ImageNet top-1 accuracy at lower FLOPs.
Calibrating Transformers via Sparse Gaussian Processes
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing, speech recognition and computer vision. Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It replaces the scaled dot-product operation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection.
fields
cs.CV 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Compact Vision Transformer by Reduction of Kernel Complexity
KCR-Transformer replaces standard transformer blocks with channel-pruned MLPs plus a truncated-nuclear-norm regularizer, reporting better ImageNet top-1 accuracy at lower FLOPs.