Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training

Hua Li

REVIEW 2 major objections 2 minor 3 references

Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training

T0 review · 2 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LPG adaptively perturbs logit-level gradients at the class level for category-aware neural network training.

desk verdict The paper unifies gradient perturbation methods under one view and adds a class-adaptive learned version, but the design hinges on an untested conjecture about norm changes acting as positive or negative augmentation. read the letter →

arxiv 2605.29494 v1 pith:HSD5ES37 submitted 2026-05-28 cs.LG

Hua Li This is my paper

classification cs.LG

keywords gradientperturbationadaptivetrainingcategory-awarelearninglogitgradientslong-tailclassificationnoisylabelPAC-Bayesianboundsplug-inmodule

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a unified framework showing that existing techniques like SAM, gradient clipping, and noise injection all impose particular forms of gradient perturbation during backpropagation. It draws an analogy to logit perturbation learning and conjectures that increasing a class's gradient norm acts as positive augmentation while decreasing it acts as negative augmentation. From this, the authors derive LPG, a method that learns class-specific perturbations of logit gradients to produce category-aware updates. Experiments across balanced, long-tailed, and noisy-label settings show LPG outperforming prior approaches and functioning as a compatible plug-in module. PAC-Bayesian analysis supplies generalization bounds tied to the size of the introduced gradient perturbations.

What carries the argument

Learning to Perturb Gradients (LPG), which learns class-specific scaling factors for logit gradients to implement the positive/negative augmentation conjecture.

What would settle it

A controlled experiment in which LPG shows no accuracy gain over strong baselines on at least three distinct datasets or in which adding LPG to SAM yields no further improvement.

Watch

Extended reading notes

Core claim

Gradient perturbation along the backward pass can be treated systematically; by learning to amplify or dampen logit-level gradients on a per-class basis, training becomes category-aware, yielding consistent gains on standard, long-tail, and noisy-label classification while connecting perturbation magnitude to PAC-Bayesian generalization guarantees.

Load-bearing premise

Amplifying a class gradient norm improves learning for that class while dampening it reduces overfitting to that class.

Editorial extensions

If this is right

LPG can be inserted into existing training pipelines without architectural changes.
The same class-level perturbation logic applies to long-tail and noisy-label regimes.
PAC-Bayesian bounds link the allowed perturbation size directly to generalization error.
Category-aware gradient scaling produces training dynamics distinct from uniform regularization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the augmentation conjecture holds, analogous scaling could be applied to intermediate feature gradients rather than only logits.
The plug-in nature suggests LPG might compose with non-gradient methods such as data augmentation schedules.
Category-specific gradient control could transfer to settings where class imbalance arises dynamically during training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is that this work organizes SAM, gradient clipping, and noise injection as instances of gradient perturbation, then introduces LPG to make the perturbation strength class-dependent and learned. The unification gives a cleaner way to compare these techniques, and the class-level adaptation is a direct extension beyond the fixed perturbations in prior work.

What stands out as new is the LPG rule itself, which perturbs logit-level gradients per class, plus the attempt to tie the approach to PAC-Bayesian generalization bounds. The experiments cover balanced, long-tail, and noisy-label settings and report that LPG improves over baselines and combines with them.

The framework is a reasonable organizing lens. The plug-in nature of LPG is practical for people already using other regularizers.

The clearest soft spot is the motivating conjecture: amplifying a class gradient norm is treated as positive augmentation while dampening it is negative augmentation, justified mainly by analogy to LPL. No separate check of this directional claim appears in the abstract, and the method and bounds are built on it. If the effect does not hold, the claimed category-aware benefit collapses to generic perturbation whose advantage over SAM or clipping becomes empirical only. The abstract also omits error bars, ablation details, and exact bound derivations, so those need verification.

This is for readers who work on optimizer tweaks for imbalanced or noisy classification. Someone hunting for new adaptive regularizers might extract usable ideas even if the conjecture requires more support.

Send it to peer review. The unification and the new method are coherent enough that referees can assess the evidence and the theory in the full text.

Referee Report

2 major / 2 minor

Summary. The paper establishes a unified framework interpreting existing gradient perturbation techniques (SAM, clipping, noise injection) as specific forms of perturbation along the backward pass. Analogous to Logit Perturbation Learning, it conjectures that class-wise amplification of gradient norms provides positive augmentation while dampening provides negative augmentation. It proposes Learning to Perturb Gradients (LPG), an adaptive class-level perturbation of logit gradients for category-aware training, derives PAC-Bayesian generalization connections, and reports consistent outperformance on balanced, long-tail, and noisy-label tasks, with LPG usable as a plug-in module.

Significance. If the conjecture holds and the reported gains are attributable to the proposed mechanism rather than generic perturbation, the work supplies a new, composable tool for adaptive training that is particularly relevant for imbalanced and noisy settings. The unified framework and PAC-Bayesian links, if rigorously parameter-independent, would aid understanding of perturbation methods. The plug-in compatibility is a practical strength.

major comments (2)

[Introduction] Introduction (conjecture statement): The claim that 'amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting)' is introduced by direct analogy to LPL with no supporting derivation, preliminary experiment, or ablation isolating the directional effect. This conjecture directly dictates the adaptive perturbation rule in the LPG method; if the directional effect is neutral or reversed, the category-aware motivation collapses to that of any generic gradient perturbation whose advantage over SAM/clipping must then be shown empirically alone.
[Theoretical analysis] PAC-Bayesian section: The generalization bound is presented as connecting gradient perturbation to guarantees, yet the text does not clarify whether the bound remains independent of the learned perturbation parameters or is evaluated after fitting; if the latter, the analysis risks circularity with the method design and does not provide an a-priori guarantee.

minor comments (2)

[Method] Notation for logit-level gradients should be defined once with a clear symbol (e.g., g_c for class c) and used consistently; current usage mixes descriptive phrases with symbols across sections.
[Experiments] Figure captions for the long-tail and noisy-label results should explicitly state the number of runs, whether error bars are standard deviation or standard error, and the exact baseline implementations used for comparison.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and for identifying areas where the manuscript can be strengthened. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

Referee: [Introduction] Introduction (conjecture statement): The claim that 'amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting)' is introduced by direct analogy to LPL with no supporting derivation, preliminary experiment, or ablation isolating the directional effect. This conjecture directly dictates the adaptive perturbation rule in the LPG method; if the directional effect is neutral or reversed, the category-aware motivation collapses to that of any generic gradient perturbation whose advantage over SAM/clipping must then be shown empirically alone.

Authors: The conjecture is presented as an analogy to LPL within the unified gradient-perturbation framework developed in the paper. We agree that additional empirical grounding would strengthen the motivation for the class-wise directional rule. In the revised version we will add a short preliminary ablation that measures the effect of class-wise gradient-norm amplification versus dampening on per-class accuracy and overfitting indicators, thereby providing direct support for the adaptive rule rather than relying solely on the LPL analogy. revision: yes
Referee: [Theoretical analysis] PAC-Bayesian section: The generalization bound is presented as connecting gradient perturbation to guarantees, yet the text does not clarify whether the bound remains independent of the learned perturbation parameters or is evaluated after fitting; if the latter, the analysis risks circularity with the method design and does not provide an a-priori guarantee.

Authors: The PAC-Bayesian bound is intended to be a general, parameter-independent statement that relates any gradient perturbation (including the class-wise form used by LPG) to a generalization guarantee. We acknowledge that the current exposition does not explicitly state this independence. In the revision we will rewrite the relevant paragraph to emphasize that the bound is derived before the perturbation parameters are optimized and holds for any fixed perturbation function, thereby removing any appearance of circularity. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation chain

full rationale

The paper's chain proceeds from a unified interpretive framework (existing methods as gradient perturbations), to an explicit conjecture motivated by analogy to LPL, to the LPG design, to PAC-Bayesian connections, and to experiments. No equation or claim reduces by construction to its own inputs; the conjecture is stated as an assumption rather than derived, the framework is interpretive rather than tautological, and the PAC-Bayesian analysis is presented as providing independent generalization guarantees. Self-citation to LPL is present but not load-bearing for any uniqueness theorem or fitted prediction. The derivation remains self-contained against external benchmarks and does not exhibit any of the enumerated circular patterns.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Abstract-only view limits visibility into parameters or entities; the central conjecture functions as a domain assumption without independent evidence provided here.

assumptions (1)

domain assumption Amplifying the gradient norm for a class acts as positive augmentation while dampening acts as negative augmentation
Explicitly stated as the basis for LPG design in the abstract.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training." pith.science (2026). https://pith.science/paper/HSD5ES37

@misc{pith2026260529494,
  author       = {Pith},
  title        = {Pith review of: Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSD5ES37}},
  note         = {Machine review of arXiv:2605.29494}
}

read the original abstract

Deep neural network training involves both forward propagation (from features through logits to loss) and backward propagation (from loss through gradients to parameter updates). While perturbations along the forward chain, including feature perturbation, logit perturbation, and label perturbation, have been extensively studied, the backward chain's gradient perturbation has received little systematic investigation. In this paper, we establish a unified framework for gradient perturbation, revealing that existing methods such as Sharpness-Aware Minimization (SAM), gradient clipping, and gradient noise injection can all be interpreted as imposing specific forms of gradient perturbation. Analogous to the recently proposed Logit Perturbation Learning (LPL), we conjecture that amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting). Based on these observations, we propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training. We also establish theoretical connections between gradient perturbation bounds and generalization guarantees via PAC-Bayesian analysis. Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate that LPG consistently outperforms existing methods and can be combined with them as a plug-in module.

Figures

Figures reproduced from arXiv: 2605.29494 by the authors.

**Figure 1.** Relative gradient variation ∥g˜c − g¯c∥/∥g¯c∥ across classes. (a) On balanced data, LPG applies larger perturbation to lower-accuracy classes. (b) On long-tail data, LPG adaptively amplifies tail class gradients and dampens head class gradients, while SAM, gradient clipping, and gradient noise fail to provide class-adaptive perturbation. et al., 2019] and adversarial training [Madry et al., 2018], which modify the i… view at source ↗

**Figure 2.** Long-tail classification accuracy on CIFAR-10-LT and CIFAR-100-LT [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

**Figure 3.** Test accuracy under symmetric label noise on CIFAR-10 and CIFAR [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: (a) Combination of LPG with existing methods on CIFAR-100 (WRN [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: t-SNE visualization of features learned by different methods on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: Sensitivity analysis of LPG to key hyperparameters on CIFAR-10-LT [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[1]

Simplified pac-bayesian margin bounds

David A McAllester. Simplified pac-bayesian margin bounds. InLearning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27,

2003
[2]

Adding Gradient Noise Improves Learning for Very Deep Networks

URLhttps: //openreview.net/forum?id=37nvvqkCo5. Arvind Neelakantan, Luke Vilnis, Quoc V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks.arXiv preprint arXiv:1511.06807,

work page Pith review arXiv
[3]

Large Batch Training of Convolutional Networks

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 322–330, October 2019a. Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for d...

work page Pith review arXiv

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.

[1] [1]

Simplified pac-bayesian margin bounds

David A McAllester. Simplified pac-bayesian margin bounds. InLearning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27,

2003

[2] [2]

Adding Gradient Noise Improves Learning for Very Deep Networks

URLhttps: //openreview.net/forum?id=37nvvqkCo5. Arvind Neelakantan, Luke Vilnis, Quoc V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks.arXiv preprint arXiv:1511.06807,

work page Pith review arXiv

[3] [3]

Large Batch Training of Convolutional Networks

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 322–330, October 2019a. Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for d...

work page Pith review arXiv