Towards Understanding Why Data Augmentation Improves Generalization

Jiachun Pan; Jingyang Li; Kim-Chuan Toh; Pan Zhou

REVIEW 3 major objections 4 minor 43 references

Towards Understanding Why Data Augmentation Improves Generalization

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Data augmentation improves generalization through two effects, the paper argues: partially removing semantic features and mixing in features from other images; combining both gives complementary gains.

desk verdict A coherent k-class extension of the Allen-Zhu-Li framework to a two-effect taxonomy of augmentations, but the A2 theorem explicitly predicts no clean-test gain from Mixup while the experiments show the opposite, so the paper's own validation undercuts its theory. read the letter →

arxiv 2502.08940 v1 pith:5VAWSBZR submitted 2025-02-13 cs.CV cs.LGstat.ML

Jingyang Li , Jiachun Pan , Kim-Chuan Toh , Pan Zhou This is my paper · ORCID

classification cs.CVcs.LGstat.ML

keywords dataaugmentationgeneralizationtheoryfeaturelearningpartialsemanticremovalmixingmulti-viewMixupCutMix

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks a unified explanation for why very different augmentations—random cropping, color jittering, CutOut, Mixup, CutMix, and SaliencyMix—all improve generalization. It proposes that they act through two effects: partial semantic feature removal (A1), which stops the network from relying on a single feature and forces it to learn diverse features, and feature mixing (A2), which scales down original semantic features, injects noisy features, and forces more robust features. On a multi-view data model with two discriminative features per class, the paper proves for a three-layer CNN that A1-trained models learn both class features and classify single-view test samples with margin, that A2-trained models have a strictly larger feature-learning signal and survive noisier test data, and that A3 (both effects) achieves both properties. The authors note that the analysis is restricted to CNN image classification and to idealized constructions of the two effects rather than direct measurements of each augmentation.

What carries the argument

The load-bearing object is the feature-learning indicator $\Phi^{(t)}_{i,l} = \sum_{r\in[m]} [\langle w^{(t)}_{i,r}, v_{i,l}\rangle]_+$, the total positive alignment between the class-$i$ convolution kernels and the $l$-th discriminative feature of that class. The argument works by encoding each augmentation as a change in the distribution of feature patch-coefficient sums $\sum_{p\in\mathcal{P}_v(X)} z_p$: A1 reduces one semantic feature to $[C_1, O(1)]$, A2 multiplies semantic features by $1-C_2$ and raises noisy features by $C_3$, and A3 does both, each with probability at least $1/\mathrm{polylog}(k)$. Under this encoding, the gradient-descent analysis follows the same induction as the paper's vanilla supervised-learning analysis, with the augmented samples acting as extra single-view or extra-hard multi-view training points; the final $\Omega(\log k)$ margins in the theorems come from comparing the resulting $\Phi$ values with those of vanilla training.

What would settle it

Construct or select images with known orthogonal semantic feature vectors, apply random cropping, CutOut, Mixup, and CutMix, and measure the per-feature patch-coefficient sums in the augmented images; if cropping rarely reduces one feature to $C_1\in(0,0.4)$ while leaving the other near $1$, or Mixup's semantic/noisy scale changes do not satisfy $C_2+C_3<0.6$, the predicted accuracy gaps on single-view and noisy test sets should fail for those methods.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that the success of these augmentations reduces to two changes in the feature content of training images. With probability $\pi_1$, A1 removes one of the two semantic features of a class by cutting its patch-coefficient sum to $[C_1, O(1)]$; with probability $\pi_2$, A2 scales original semantic features down by $1-C_2$ and scales noisy features up to $C_3$; A3 does both. Theorems 4.3, 4.6, and 4.8 then show that, after $T=\mathrm{poly}(k)/\eta$ gradient-descent steps on a three-layer CNN, the feature-learning indicators $\Phi^{(t)}_{i,l}$ follow different laws: with A1 both per-class features reach $\Omega(\log k)$ whereas vanilla training leaves one feature at $1/\mathrm{polylog}(k)$; with A2 the total indicator is at least $\frac{0.6}{0.6-C_2-C_3}$ times the vanilla value; with A3 both bounds hold. The paper converts these bounds into generalization statements: A1-trained networks classify single-view test samples with an $\Omega(\log k)$ margin, A2-trained networks classify noisy test samples that vanilla networks fail, and A3-trained networks do both.

Load-bearing premise

The load-bearing premise is that real augmentations are faithfully described by the three idealized effect models—A1 cutting one semantic feature to $[C_1, O(1)]$, A2 scaling semantic features by $1-C_2$ and noisy features up by $C_3$, A3 doing both, each with probability at least $1/\mathrm{polylog}(k)$—so that if random cropping shrinks both features together or Mixup's effect is not that uniform scaling, the theorems no longer apply to the named methods.

Editorial extensions

If this is right

Any augmentation that behaves like A1 should make the CNN learn both per-class discriminative features and classify single-view images almost perfectly, where vanilla supervised learning stays near chance on those images.
Any augmentation that behaves like A2 should leave clean test accuracy comparable to vanilla while strictly improving accuracy on noisy test inputs, with the gain factor controlled by $0.6/(0.6-C_2-C_3)$.
Any augmentation that behaves like A3 should inherit both effects, and the proof gives a reason for the combination to be more than additive: the allowed range of $C_1$ is widened and the robust-feature lower bound in Theorem 4.8 exceeds that in Theorem 4.6.
The same mechanism covers the paper's experiments: random cropping, color jittering, and CutOut form the A1 group, Mixup the A2 group, and CutMix, SaliencyMix, and random cropping plus Mixup the A3 group, with test accuracy ordered accordingly on CIFAR-100 and Tiny-ImageNet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The constants $\pi_1,\pi_2,\pi_3,C_1,C_2,C_3$ are never estimated from the real augmentations, so I read the framework as a conditional explanation: measuring per-feature coefficient sums on actual augmented images would determine which named methods fall inside the assumptions and where the predictions should break.
A testable design rule follows from the noisy-data case: because the $\Omega(\log k)$ margin on the noisy distribution appears when $0.6-C_2-C_3$ is small, tuning a mixing augmentation so that the post-mixing semantic scale nearly matches the noisy scale should maximize robustness gains.
The vocabulary of removal and mixing could be carried to augmentations in other modalities, such as deleting tokens in text or mixing waveforms in audio, but the proof is for image patches and CNNs, so such transfers are extrapolations rather than consequences of the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper deserves a serious referee, but its own experiments undercut the A2 story. What is new is the organizing taxonomy: partial semantic feature removal (A1) encourages diverse feature learning, feature mixing (A2) encourages robust features, and the combination (A3) yields both. The k-class theorems (4.3, 4.6, 4.8) go beyond the existing binary analyses of single augmentations, and the proofs are detailed and, as the appendix says, largely run the Allen-Zhu-Li machinery with modified constants. The A1 result in particular is clean and plausible: if an augmentation probabilistically removes one semantic feature, the model must learn both features to minimize the augmented loss, and the generalization to single-view test data follows. Credit where it is earned: this is a real, formal organizing result for the feature-learning theory of augmentation.

The biggest problem is not the unmeasured constants C1, C2, C3, pi — that is a standard weakness in this literature. It is the internal contradiction between Theorem 4.6(b) and the experiments used to validate A2. The theorem says plainly that for clean test data D, SL with A2 has the same accuracy as vanilla SL; improvement appears only on the noisy distribution D_noisy. Yet Tables 1 and 2 report Mixup improving clean accuracy by 3-4 points on CIFAR-100 and Tiny-ImageNet. The experiments never evaluate on a noisy test distribution, so they validate exactly the opposite of what the A2 theorem predicts. Even if the constants were derived from Mixup's alpha, the theorem would still say no clean-data gain. The paper does not flag or reconcile this; it states that Mixup 'shows improved generalization' as if that were support.

The A3 and A1 experimental trends (CutMix, CutOut, random cropping) are consistent with the theory's qualitative predictions, and those parts of the paper hold up better. But the claimed unified explanation for Mixup specifically is not supported by the reported evidence.

Who this is for: ML theory readers working on feature-learning explanations of augmentation. It deserves a serious referee, but the revision must either test the noisy-data prediction directly, reclassify Mixup as A3, or change Theorem 4.6(b) to admit clean gains. As it stands, the central claim about Mixup is contradicted by the paper's own tables.

Referee Report

3 major / 4 minor

Summary. The paper proposes a unified theoretical framework for why data augmentation improves generalization. It models two idealized effects: partial semantic feature removal (A1), which is claimed to promote diverse feature learning, and feature mixing (A2), which is claimed to promote robust feature learning; their combination (A3) is claimed to yield complementary benefits. Using the multi-view/single-view data model of Allen-Zhu and Li (2023) and a three-layer CNN, the paper proves theorems about feature-learning indicators under Assumptions 4.1, 4.4, and 4.7/A.5: Theorem 4.3 states that A1 makes the network learn both semantic features per class and improves single-view test accuracy; Theorem 4.6 states that A2 yields a larger feature-learning indicator and improves accuracy on a noisy test distribution while leaving clean-distribution accuracy unchanged; Theorem 4.8 states that A3 achieves both diverse and robust feature learning. Experiments on CIFAR-100 and Tiny-ImageNet with VGG-16 and DenseNet-121 report clean test accuracy for vanilla training and for random cropping, color jittering, CutOut, Mixup, CutMix, SaliencyMix, and cropping plus Mixup.

Significance. If the assumptions are accepted and the mapping from the named augmentations to A1/A2/A3 were established, the paper would offer a useful unifying perspective on a wide class of augmentation methods, with explicit predictions about feature-learning indicators and about robustness to noisy test inputs. The proofs are detailed and appear internally consistent within the stated model, and the paper builds carefully on a known feature-learning framework. The main conceptual strength is the explicit statement that A1 and A2 address different failure modes and that A3 combines them. However, the load-bearing assumptions introduce free constants and probabilities that are never connected to the actual augmentation mechanisms, and the experimental section tests clean accuracy only, which is not the distribution where the A2 theorem predicts an improvement. In its current form, the paper overstates the degree to which the experiments validate the theory.

major comments (3)

[§5.1, Tables 1-2; §4.2, Theorem 4.6(b)] Theorem 4.6(b) states that for the clean data distribution D, SL with A2 has the same test accuracy as vanilla SL, with improvement appearing only on the noisy distribution D_noisy (Definition A.3). Yet the experimental section reports Mixup improving clean test accuracy in every setting, e.g., 67.41 vs. 63.69 for VGG-16 on CIFAR-100 and 54.93 vs. 50.65 on Tiny-ImageNet, and the text in §5.1 attributes this to the feature-mixing effect. The reported experiments therefore do not validate the A2 mechanism; they are in direct tension with the paper's own formal prediction. To support the theory, the authors should either report results on a noisy test distribution as defined in the paper, or explain why the clean-accuracy gains observed for Mixup lie outside the scope of the A2 theorem.
[Assumptions 4.1, 4.4, 4.7 and A.5] The central theorems are conditional on abstract augmentation effects whose constants C1, C2, C3 and probabilities π1, π2, π3 are never derived from the named augmentation methods. For example, no relationship is given between Mixup's Beta parameter and C2/C3, between the CutMix mask distribution and C1/C2, or between random-crop scale and π1. Consequently, Theorems 4.3, 4.6, and 4.8 establish results for an abstract operation that may or may not correspond to random cropping, Mixup, or CutMix. The paper's claim that it explains the success of these specific methods requires either deriving the constants for representative augmentations or explicitly reframing the results as conditional on uninstantiated operations.
[§5, experimental protocol] The experimental section reports single accuracy numbers without error bars, number of seeds, or training hyperparameters (epochs, learning rate, batch size, augmentation strengths). Because the reported gains are often only a few percentage points, the reader cannot assess whether the differences are significant or whether the results are reproducible. At minimum, the authors should report mean and standard deviation over multiple seeds and provide the exact training settings.

minor comments (4)

[Abstract and §3.2] The abstract contains a grammatical error, 'individual feature' should be 'individual features', and the text after Assumption 4.1 states 'we does not require assumption', which should read 'we do not require an assumption'.
[Equation (1)] The quantity logiti(F,X) is defined as a softmax probability, not a logit or log-odds. Consider renaming it 'softmax probability' to avoid confusion with standard terminology.
[§4.2, Theorem 4.6] In the paragraph following Theorem 4.6, 'vanilla L without A2' should read 'vanilla SL without A2'.
[Definition A.1] The terms 'left-close' and 'right-close' are used without definition; please define them or replace them with explicit inequalities.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theorems are conditional on explicitly stated idealized augmentation effects, and the constants are not fitted predictions.

full rationale

The paper's formal results (Theorems 4.3, 4.6, 4.8) are conditional theorems: given Assumptions 4.1, 4.4, and 4.7, the proofs derive feature-learning and generalization bounds for the three-layer CNN. The assumptions describe idealized effects of augmentation on patch coefficient sums (e.g., one semantic feature reduced to scale [C1,O(1)] under A1; semantic features scaled by 1-C2 and noisy features increased by C3 under A2). These assumptions are not defined in terms of the conclusions they are used to prove: 'diverse feature learning' (learning both vi,1 and vi,2) and 'robust feature learning' (larger Φ) are distinct from the assumed input-side scale changes. Nor are C1, C2, C3, π1, π2, π3 fitted to the reported accuracies; they are free parameters of the idealized model. The proof framework follows Allen-Zhu & Li (2023), which is an external prior work, not the present authors' self-citation, and the vanilla-SL comparisons are imported from that external source. The main concern in the reader's take is that the assumptions already encode the mechanism, so the theorems formalize rather than independently predict the effects; that is a standard assumption-based theory structure, not a circular reduction of output to input by construction. The skeptic's internal-mismatch point is also real but is a validity/evidential problem, not circularity: Theorem 4.6(b) predicts no clean-test-accuracy gain for A2, while Tables 1-2 show Mixup improving clean accuracy, so the experiments do not validate that particular theoretical prediction. That mismatch does not make the derivation circular. Because no step exhibits a fitted parameter renamed as a prediction, a self-citation chain carrying the load, or a definition of the conclusion into the premise, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The ledger shows that the paper's conclusions depend on three unmeasured constants and three unmeasured probabilities for the augmentation effects, on top of the entire multi-view data-distribution and proof framework adopted from Allen-Zhu and Li. No independent measurement or falsifiable handle is provided for these parameters.

free parameters (4)

C1 = no fixed value, assumed in (0, 0.4)
Scale of the remaining semantic feature under A1; the theorems' regimes and the stated relaxation C1 < 0.4 + C2 + C3 depend on it, but it is never estimated from real augmentations.
C2 = no fixed value, assumed positive with C2 + C3 < 0.6
Assumed scale reduction of semantic features under feature mixing; the improvement factor in Eq (12) is a function of C2.
C3 = no fixed value, assumed positive with C2 + C3 < 0.6
Assumed scale increase of noisy features under feature mixing; appears in the margin 0.6 - C2 - C3 that controls the robustness prediction.
pi1, pi2, pi3 = no fixed values, assumed >= 1/polylog(k)
Probabilities that the augmentation effects occur; they enter the training-loss decomposition in Eqs (20), (24), (28) and need to be large enough for the error bounds.

assumptions (5)

domain assumption Multi-view/single-view data distribution from Allen-Zhu and Li 2023 (Definition 3.1 and A.1) with two orthogonal features per class, sparsity s = polylog(k), and mixing probability mu = 1/poly(k)
The entire analysis is couched in this synthetic distribution; it is borrowed from prior work and not re-derived.
domain assumption The Allen-Zhu and Li 2023 proof framework: smoothed ReLU (Eq 19), Gaussian initialization with sigma0, the 'lottery winning' set, their Induction Hypotheses C.1/C.3, and Claims D.4-D.6 are correct and applicable
Appendices F and G state that the proof follows the same framework and cite the prior induction hypothesis instead of reproducing it; any gap there inherits into this paper.
ad hoc to paper Assumption 4.1 (and A.5 for A3): A1 removes one semantic feature with probability pi1 while preserving the other at scale [C1, O(1)]
This is the paper's own model of augmentation; it is not derived from or validated against random cropping, color jittering, or CutOut.
ad hoc to paper Assumption 4.4: A2 scales down semantic features by C2 and scales up noisy features by C3 with probability pi2
The paper's model of Mixup-like blending; the constants are unmeasured.
ad hoc to paper Assumption A.5: A3 combines the two effects with conditions on C1, C2, C3
Postulated to capture CutMix/SaliencyMix; the link to the actual mask distribution is not quantitative.

invented entities (3)

Partial semantic feature removal effect (A1)
purpose: Formalizes how cropping, jittering, and CutOut improve generalization by converting multi-view training samples into single-view ones
The effect is assumed with parameters (pi1, C1); Figure 2 illustrates it visually, but no measurement of the scale reduction in real images is given.
Feature mixing effect (A2)
purpose: Formalizes how Mixup improves robustness by reducing semantic feature scale and increasing noisy feature scale
The effect is assumed with parameters (pi2, C2, C3), and the connection to the Beta distribution in Mixup is qualitative only.
Combined effect (A3)
purpose: Formalizes how CutMix and SaliencyMix combine removal and mixing
Assumption A.5 is a hybrid of the previous two, with no direct validation from CutMix experiments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Understanding Why Data Augmentation Improves Generalization." pith.science (2026). https://pith.science/paper/5VAWSBZR

@misc{pith2026250208940,
  author       = {Pith},
  title        = {Pith review of: Towards Understanding Why Data Augmentation Improves Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VAWSBZR}},
  note         = {Machine review of arXiv:2502.08940}
}

read the original abstract

Data augmentation is a cornerstone technique in deep learning, widely used to improve model generalization. Traditional methods like random cropping and color jittering, as well as advanced techniques such as CutOut, Mixup, and CutMix, have achieved notable success across various domains. However, the mechanisms by which data augmentation improves generalization remain poorly understood, and existing theoretical analyses typically focus on individual techniques without a unified explanation. In this work, we present a unified theoretical framework that elucidates how data augmentation enhances generalization through two key effects: partial semantic feature removal and feature mixing. Partial semantic feature removal reduces the model's reliance on individual feature, promoting diverse feature learning and better generalization. Feature mixing, by scaling down original semantic features and introducing noise, increases training complexity, driving the model to develop more robust features. Advanced methods like CutMix integrate both effects, achieving complementary benefits. Our theoretical insights are further supported by experimental results, validating the effectiveness of this unified perspective.

Figures

Figures reproduced from arXiv: 2502.08940 by the authors.

**Figure 1.** Visualization of pretrained ResNet-50 (He et al., 2016) using Grad-CAM on airplane images from ImageNet. Allen-Zhu & Li, 2022; 2023; Jelassi & Li, 2022; Huang et al., 2023; Chen et al., 2024). For instance, Allen-Zhu & Li (2023) investigated the mechanisms by which ensemble methods and knowledge distillation enhance model generalization. Building on the analytical framework proposed by Allen-Zhu & Li (2023), this w… view at source ↗

**Figure 2.** Visualization of data augmentation effects on ImageNet images. (a) Partial semantic feature removal (A1): original image (left) and augmented version (right) are shown for each pair. (b) Feature mixing (A2): original images (left and middle) blended into the augmented version (right). (c) Combined effect (A3): original images (left and middle) together create the augmented version (right). iteration. A larger Φ (t) … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 22 canonical work pages

[1]

and Li, Y

Allen-Zhu, Z. and Li, Y. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pp.\ 977--988. IEEE, 2022

work page 2021
[2]

and Li, Y

Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations, 2023

2023
[3]

Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks

Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp.\ 322--332. PMLR, 2019

work page 2019
[4]

Bishop, C. M. Training with noise is equivalent to tikhonov regularization. Neural computation, 7 0 (1): 0 108--116, 1995

1995
[5]

Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005
[6]

Benign overfitting in two-layer convolutional neural networks

Cao, Y., Chen, Z., Belkin, M., and Gu, Q. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35: 0 25237--25250, 2022

work page 2022
[7]

On mixup regularization

Carratino, L., Ciss \'e , M., Jenatton, R., and Vert, J.-P. On mixup regularization. Journal of Machine Learning Research, 23 0 (325): 0 1--31, 2022

work page 2022
[8]

Chen, S., Dobriban, E., and Lee, J. H. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21 0 (245): 0 1--71, 2020

work page 2020

Show all 43 references

[9]

Why does sharpness-aware minimization generalize better than sgd? Advances in neural information processing systems, 36, 2024

Chen, Z., Zhang, J., Kou, Y., Chen, X., Hsieh, C.-J., and Gu, Q. Why does sharpness-aware minimization generalize better than sgd? Advances in neural information processing systems, 36, 2024

2024
[10]

Provably learning diverse features in multi-view data with midpoint mixup

Chidambaram, M., Wang, X., Wu, C., and Ge, R. Provably learning diverse features in multi-view data with midpoint mixup. In International Conference on Machine Learning, pp.\ 5563--5599. PMLR, 2023

2023
[11]

A kernel theory of modern data augmentation

Dao, T., Gu, A., Ratner, A., Smith, V., De Sa, C., and R \'e , C. A kernel theory of modern data augmentation. In International conference on machine learning, pp.\ 1528--1537. PMLR, 2019

2019
[12]

Imagenet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

2009
[13]

and Taylor, G

DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017

2017 arXiv
[14]

An image is worth 16x16 words: Transformers for image recognition at scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

2010 arXiv
[15]

and Sun, Y

Hanin, B. and Sun, Y. How data augmentation affects optimization for linear regression. Advances in neural information processing systems, 34: 0 8095--8105, 2021

2021
[16]

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

2016
[17]

Masked autoencoders are scalable vision learners

He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

2022
[18]

Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4700--4708, 2017

2017
[19]

Understanding convergence and generalization in federated learning through feature learning theory

Huang, W., Shi, Y., Cai, Z., and Suzuki, T. Understanding convergence and generalization in federated learning through feature learning theory. In The Twelfth International Conference on Learning Representations, 2023

2023
[20]

and Li, Y

Jelassi, S. and Li, Y. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning, pp.\ 9965--10040. PMLR, 2022

2022
[21]

Learning multiple layers of features from tiny images

Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

2009
[22]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

2012
[23]

H., and Kang, J

Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36 0 (4): 0 1234--1240, 2020

2020
[24]

and Yuan, Y

Li, Y. and Yuan, Y. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems, 30, 2017

2017
[25]

and Yun, C

Oh, J. and Yun, C. Provable benefit of cutout and cutmix for feature learning. arXiv preprint arXiv:2410.23672, 2024

2024 arXiv
[26]

A unified analysis of mixed sample data augmentation: A loss function perspective

Park, C., Yun, S., and Chun, S. A unified analysis of mixed sample data augmentation: A loss function perspective. Advances in Neural Information Processing Systems, 35: 0 35504--35518, 2022

2022
[27]

Does data augmentation lead to positive margin? In International Conference on Machine Learning, pp.\ 5321--5330

Rajput, S., Feng, Z., Charles, Z., Loh, P.-L., and Papailiopoulos, D. Does data augmentation lead to positive margin? In International Conference on Machine Learning, pp.\ 5321--5330. PMLR, 2019

2019
[28]

A., Stimberg, F., Wiles, O., and Mann, T

Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, O., and Mann, T. A. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34: 0 29935--29948, 2021

2021
[29]

R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp.\ 618--626, 2017

2017
[30]

Data augmentation as feature manipulation

Shen, R., Bubeck, S., and Gunasekar, S. Data augmentation as feature manipulation. In International conference on machine learning, pp.\ 19773--19808. PMLR, 2022

2022
[31]

and Khoshgoftaar, T

Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of big data, 6 0 (1): 0 1--48, 2019

2019
[32]

and Zisserman, A

Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

2014 arXiv
[33]

Dropout: a simple way to prevent neural networks from overfitting

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 0 (1): 0 1929--1958, 2014

1929
[34]

Saliencymix: A saliency guided data augmentation strategy for better regularization

Uddin, A., Monira, M., Shin, W., Chung, T., Bae, S.-H., et al. Saliencymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.01791, 2020

2006 arXiv
[35]

C., Murino, V., and Savarese, S

Volpi, R., Namkoong, H., Sener, O., Duchi, J. C., Murino, V., and Savarese, S. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018

2018
[36]

and Li, Y

Wen, Z. and Li, Y. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pp.\ 11112--11122. PMLR, 2021

2021
[37]

and Li, Y

Wen, Z. and Li, Y. The mechanism of prediction head in non-contrastive self-supervised learning. Advances in Neural Information Processing Systems, 35: 0 24794--24809, 2022

2022
[38]

J., Chun, S., Choe, J., and Yoo, Y

Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 6023--6032, 2019

2019
[39]

Understanding deep learning (still) requires rethinking generalization

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64 0 (3): 0 107--115, 2021

2021
[40]

mixup: Beyond empirical risk minimization

Zhang, H. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017

2017 arXiv
[41]

How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.04819, 2020

Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.04819, 2020

2010 arXiv
[42]

The benefits of mixup for feature learning

Zou, D., Cao, Y., Li, Y., and Gu, Q. The benefits of mixup for feature learning. In International Conference on Machine Learning, pp.\ 43423--43479. PMLR, 2023

2023
[43]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

and Li, Y

Allen-Zhu, Z. and Li, Y. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pp.\ 977--988. IEEE, 2022

work page 2021

[2] [2]

and Li, Y

Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations, 2023

2023

[3] [3]

Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks

Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp.\ 322--332. PMLR, 2019

work page 2019

[4] [4]

Bishop, C. M. Training with noise is equivalent to tikhonov regularization. Neural computation, 7 0 (1): 0 108--116, 1995

1995

[5] [5]

Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005

[6] [6]

Benign overfitting in two-layer convolutional neural networks

Cao, Y., Chen, Z., Belkin, M., and Gu, Q. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35: 0 25237--25250, 2022

work page 2022

[7] [7]

On mixup regularization

Carratino, L., Ciss \'e , M., Jenatton, R., and Vert, J.-P. On mixup regularization. Journal of Machine Learning Research, 23 0 (325): 0 1--31, 2022

work page 2022

[8] [8]

Chen, S., Dobriban, E., and Lee, J. H. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21 0 (245): 0 1--71, 2020

work page 2020