REVIEW 7 cited by
Swin Transformer V2: Scaling Up Capacity and Resolution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.
Forward citations
Cited by 7 Pith papers
-
Towards White-Box Deep Wireless Sensing
RF-CRATE derives a fully complex-valued white-box transformer for RF sensing from the sparse rate reduction principle and shows it matches black-box baselines across five datasets.
-
Machine Learning from Explanations
A two-stage optimization pipeline that alternates label loss with a KL divergence between feature maps of masked and unmasked inputs improves accuracy and robustness in small-data classification.
-
Mahalanobis++: Improving OOD Detection via Feature Normalization
L2-normalizing pre-logit features improves Mahalanobis-based out-of-distribution detection across 44 ImageNet models, reducing the average false positive rate by 7.6 percentage points.
-
PEAR: Equal Area Weather Forecasting on the Sphere
A transformer weather model operating natively on the equal-area HEALPix grid beats an equiangular-grid counterpart at longer lead times with 2.6x fewer parameters.
-
Beyond Pretty Pictures: Combined Single- and Multi-Image Super-resolution for Sentinel-2 Images
SEN4X, a hybrid single- and multi-image super-resolution network, lifts Sentinel-2 imagery to 2.5 m and improves land-cover classification accuracy in Hanoi over SISR, MISR, and stacked-input baselines.
-
Multi-Task Learning for Heterogeneous Prediction from Video Game State with Transfer Learning
On a large World of Tanks dataset, a shared multi-task model with equal weighting or PCGrad outperforms single-task models on average, and task/map pre-training helps most in low-data regimes.
-
Transfer Learning and Mixup for Fine-Grained Few-Shot Fungi Classification
A transfer learning pipeline with feature-level mixup and weighted sampling outperformed competition baselines on few-shot fungi classification.
Discussion (0). Sign in to comment.