REVIEW 14 cited by
Swin Transformer V2: Scaling Up Capacity and Resolution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.
Forward citations
Cited by 14 Pith papers
-
Towards White-Box Deep Wireless Sensing
RF-CRATE derives a fully complex-valued white-box transformer for RF sensing from the sparse rate reduction principle and shows it matches black-box baselines across five datasets.
-
Machine Learning from Explanations
A two-stage optimization pipeline that alternates label loss with a KL divergence between feature maps of masked and unmasked inputs improves accuracy and robustness in small-data classification.
-
Mahalanobis++: Improving OOD Detection via Feature Normalization
L2-normalizing pre-logit features improves Mahalanobis-based out-of-distribution detection across 44 ImageNet models, reducing the average false positive rate by 7.6 percentage points.
-
PEAR: Equal Area Weather Forecasting on the Sphere
A transformer weather model operating natively on the equal-area HEALPix grid beats an equiangular-grid counterpart at longer lead times with 2.6x fewer parameters.
-
Beyond Pretty Pictures: Combined Single- and Multi-Image Super-resolution for Sentinel-2 Images
SEN4X, a hybrid single- and multi-image super-resolution network, lifts Sentinel-2 imagery to 2.5 m and improves land-cover classification accuracy in Hanoi over SISR, MISR, and stacked-input baselines.
-
ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling
A 10-billion-parameter climate downscaling model trained on 65,536 GPUs at up to 4.1 exaFLOPS, with R2 0.98 to 0.99 at 7 km, but with token-count and validation caveats.
-
Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries
On the FlowBench lid-driven cavity benchmark, vision-transformer foundation models outperform neural operators in data-limited regimes, but all models generalize poorly to out-of-range Reynolds numbers and geometry ge...
-
Multi-Task Learning for Heterogeneous Prediction from Video Game State with Transfer Learning
On a large World of Tanks dataset, a shared multi-task model with equal weighting or PCGrad outperforms single-task models on average, and task/map pre-training helps most in low-data regimes.
-
Transfer Learning and Mixup for Fine-Grained Few-Shot Fungi Classification
A transfer learning pipeline with feature-level mixup and weighted sampling outperformed competition baselines on few-shot fungi classification.
-
Unified Local and Global Attention Interaction Modeling for Vision Transformers
Adding local and global token interactions before self-attention, via an aggressive convolution-pooling block and a concept-attention block, improves RetinaNet object detection mAP over non-pretrained ViT, Swin, and D...
-
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
-
Safety Monitoring of Machine Learning Perception Functions: a Survey
A survey that organizes research on runtime safety monitors for ML perception into threat identification, requirements, detection, reaction, and evaluation, and lists open challenges.
-
Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends
A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.
-
Combining physics-based and data-driven models: advancing the frontiers of research with Scientific Machine Learning
A review that maps the field of Scientific Machine Learning and presents the authors' own cardiac simulation program as its flagship application.
Discussion (0). Continue with ORCID to comment.