RECORD
PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks
Machine Learning · Submitted 14 Jun 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our experiments on CIFAR10/100, SVHN, Tiny-ImageNet, and ImageNet using ResNet architectures including PreActResnet18/34, WRN-28-10, ResNet101/152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data. A recent class of methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. We propose PatchUp, a hidden state block-level regularization technique for Convolutional Neural Networks (CNNs), that is applied on selected contiguous blocks of feature maps from a random pair of samples. Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches. Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. Our experiments on CIFAR10/100, SVHN, Tiny-ImageNet, and ImageNet using ResNet architectures including PreActResnet18/34, WRN-28-10, ResNet101/152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. We also show that PatchUp can provide a better generalization to deformed samples and is more robust against adversarial attacks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data. | background 1.00 | 0.05 | 0.02 | asserted 0.66 |
| s1 | A recent class of methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. | background 1.00 | 0.03 | 0.02 | asserted 1.00 |
| s2 | We propose PatchUp, a hidden state block-level regularization technique for Convolutional Neural Networks (CNNs), that is applied on selected contiguous blocks of feature maps from a random pair of samples. | contribution 0.75 | 0.83 | 0.10 | asserted 0.99 |
| s3 | Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches. | result 0.63 | 0.70 | 0.12 | asserted 0.78 |
| s4 | Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. | interpretation 0.73 | 0.74 | 0.05 | asserted 1.00 |
| s5 | Our experiments on CIFAR10/100, SVHN, Tiny-ImageNet, and ImageNet using ResNet architectures including PreActResnet18/34, WRN-28-10, ResNet101/152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. | result 1.00 | 0.97 | 0.73 | asserted 1.00 |
| s6 | We also show that PatchUp can provide a better generalization to deformed samples and is more robust against adversarial attacks. | result 1.00 | 0.96 | 0.25 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data. A recent class of methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. We propose PatchUp, a hidden state block-level regularization technique for Convolutional Neural Networks (CNNs), that is applied on selected contiguous blocks of feature maps from a random pair of samples. Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches. Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. Our experiments on CIFAR10/100, SVHN, Tiny-ImageNet, and ImageNet using ResNet architectures including PreActResnet18/34, WRN-28-10, ResNet101/152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. We also show that PatchUp can provide a better generalization to deformed samples and is more robust against adversarial attacks.
Human review
References
How the literature uses it
15 arXiv papers cite this in 21 sentences
Papers citing it · 11 sentences without a confident label
Contests it 1 paper
-
However, these hand-crafted methods [ref] cannot guarantee mixed samples containing target objects and might cause the label mismatch problem.
AutoMix: Unveiling the Power of Mixup for Stronger Classifiers · Introduction · contest 0.74
Cites it as background 8 papers
-
Mixup [ref] and its numerous variants have seen remarkable success in supervised learning problems, as well as other problems such as semi-supervised learning [ref] , unsupervised learning using autoencoders [ref] , adversarial learning [ref] , graph-based learning [ref] , computer vision [ref] , natural language [ref] and speech [ref] .
MixupE: Understanding and Improving Mixup from Directional Derivative Perspective · Related Work · 0.93
-
Especially, mixed sample data augmentation (MSDA) [ref] has become a standard technique to train a strong deep model by synthesizing a mixed sample from multiple (usually two) samples by combining both of their sample values and labels in a linear combination [ref] or a cut-and-paste manner [ref] .
A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective · Introduction · 0.90
-
Mixup [ref] and its numerous variants [ref] have seen remarkable success in supervised learning problems, as well other problems such as semi-supervised learning [ref] , unsupervised learning using autoencoders [ref] , adversarial learning [ref] , graph-based learning [ref] , computer vision [ref] , natural language [ref] and speech [ref] .
Towards Domain-Agnostic Contrastive Learning · Related Work · 0.87
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.