REVIEW 4 cited by
Vision Transformer for Small-Size Datasets
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, the Vision Transformer (ViT), which applied the transformer structure to the image classification task, has outperformed convolutional neural networks. However, the high performance of the ViT results from pre-training using a large-size dataset such as JFT-300M, and its dependence on a large dataset is interpreted as due to low locality inductive bias. This paper proposes Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA), which effectively solve the lack of locality inductive bias and enable it to learn from scratch even on small-size datasets. Moreover, SPT and LSA are generic and effective add-on modules that are easily applicable to various ViTs. Experimental results show that when both SPT and LSA were applied to the ViTs, the performance improved by an average of 2.96% in Tiny-ImageNet, which is a representative small-size dataset. Especially, Swin Transformer achieved an overwhelming performance improvement of 4.08% thanks to the proposed SPT and LSA.
Forward citations
Cited by 4 Pith papers
-
Multimodal learning enables instant ionizing radiation alerts on unmodified mobile phones for real-world emergency response
A multimodal CNN+MLP pipeline on an unmodified iPhone camera detects and estimates hazardous radiation dose rates (25-280 mRem/h) with 86-96% accuracy in six seconds, and low-level sources (~0.6 mRem/h) with 87% accur...
-
CoSwin: Convolution Enhanced Hierarchical Shifted Window Attention For Small-Scale Vision
CoSwin fuses a two-layer 3x3 convolution into each Swin attention block and reports higher accuracy on small-scale image classification benchmarks.
-
DuoFormer: Leveraging Hierarchical Representations by Local and Global Attention Vision Transformer
A hybrid CNN-transformer model with scale-wise local and global attention reports accuracy gains over several baselines on kidney cancer tissue classification.
-
CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edge
A compact spatio-temporal network mixing convolutions and linear-complexity temporal attention reaches strong accuracy on UCF101, HMDB51, and Kinetics400 with a 7M-parameter model and float16 training.
Discussion (0). Continue with ORCID to comment.