pith. sign in

arxiv: 2604.27987 · v1 · submitted 2026-04-30 · 💻 cs.LG

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Nghia Bui , Lijing Wang This is my paper

Pith reviewed 2026-05-07 05:54 UTC · model grok-4.3

classification 💻 cs.LG
keywords fine-tuninggradient descenttraining stabilitypretrained modelsclassificationoptimizationcollapsed statesgradient scaling
0
0 comments X

The pith

Dynamic scaled gradient descent stabilizes fine-tuning by scaling down gradients from correctly classified examples to avoid collapsed training states.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes dynamic scaled gradient descent to fix instability during fine-tuning of pretrained models on sparse or imbalanced classification datasets. It argues that collapsed states occur when gradients from different examples cancel each other out, trapping the model in poor performance. The method applies a dynamic scaler that reduces the size of gradients contributed by correctly classified examples while leaving harder ones largely untouched. This produces more consistent training runs and higher final accuracy than standard fine-tuning or prior stabilization techniques. The approach is tested across multiple benchmarks and large models, showing lower variance and better results.

Core claim

By directly modifying per-example gradients to down-scale those from correct classifications using a dynamic factor, optimization avoids cancellation-induced collapse, yielding lower performance variance and higher accuracy than existing fine-tuning methods on benchmark datasets and large pretrained models.

What carries the argument

Dynamic scaled gradient descent, an optimizer that multiplies the gradient of each correctly classified training example by a time-varying scalar less than one while leaving incorrect examples unchanged.

If this is right

  • Fine-tuning runs on new sparse datasets reach higher accuracy without getting stuck.
  • Variance in final performance across repeated trainings is reduced for the same model and data.
  • The method works on large pretrained models without requiring changes to the architecture or loss.
  • Theoretical stability guarantees follow from the reduced influence of easy examples during late training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same selective scaling idea could be applied to other gradient-based methods such as Adam or to non-classification tasks like regression or generation.
  • It raises the question whether other signals besides correctness, such as loss magnitude, could serve as the basis for dynamic scaling.
  • If the dynamic scaler can be computed cheaply, the approach may become a drop-in replacement for standard optimizers in production fine-tuning pipelines.

Load-bearing premise

The primary driver of collapsed states is gradient cancellation between correct and incorrect examples, and selectively shrinking the correct ones will improve stability without slowing overall learning or creating new instabilities.

What would settle it

A controlled comparison on a standard benchmark where the dynamic scaling method produces equal or higher variance in accuracy across random seeds than plain fine-tuning or prior stabilizers.

Figures

Figures reproduced from arXiv: 2604.27987 by Lijing Wang, Nghia Bui.

Figure 1
Figure 1. Figure 1: Training failures (either collapsed state or degenerate solution) on various tasks when fine-tuning pretrained models across 10 random seeds. Each task uses the same hyperparameters, with only the random seed varying between runs. training runs, a process that is both computationally pro￾hibitive and impractical for real-time applications. Training stability is a well-documented challenge, particularly dur… view at source ↗
Figure 2
Figure 2. Figure 2: (a) and (b) are cross-cosine similarity of [CLS] token representations for 10 randomly selected examples (5 per class) when fine-tuning RoBERTa-large on COPA (a binary classification task), in a collapsed state (random seed 132) and a successful run (random seed 42), respectively. Axes list the corresponding labels. Representations in collapsed states become uniformly similar, while successful runs show hi… view at source ↗
Figure 3
Figure 3. Figure 3: compares accuracy, standard deviation, and training time across all methods using RoBERTa-large on Mul￾tiRC. We observe that although baseline methods can miti￾gate instability for some tasks, they come with significant trade-offs in efficiency, performance, or both. Single-learner baselines like LNSR, NoisyTune, and FocalLoss require no additional time or storage compared to FFT, yet they un￾derperform in… view at source ↗
Figure 4
Figure 4. Figure 4: Sensitivity analysis conducted using RoBERTa-large on COPA dataset. stable than FFT (γ = 1). This occurs because larger τ in￾corporates more signals from correctly classified examples, analogous to increasing the effective batch size, thereby boosting performance and updating stability. Hence, we set τ = 1 in all experiments. Practical Guidelines. FocalLoss and DSGD with configu￾rations like descending lin… view at source ↗
Figure 6
Figure 6. Figure 6: ViT settings. E.3. Settings & Implementations We follow (Liu et al., 2019; Hu et al., 2022) settings to finetune RoBERTa and Llama on GLUE and SuperGLUE benchmarks. All approaches are optimized using AdamW (Loshchilov & Hutter, 2017) optimizer. Baselines: • For PCGrad (Yu et al., 2020) we customize it for our task where we consider each group of training example in every iteration as an individual task in … view at source ↗
Figure 7
Figure 7. Figure 7: Gradient norm comparison for RoBERTa-large on COPA (seed 132 and seed 456). (a), (d) Original failed runs. (b), (e) Stable runs corrected by DSGD. In the failed run, cosine similarity reveals severe gradient conflicts. Negative values occur in 90% of iterations, with 30% falling below -0.5. Applying DSGD does not significantly change these directions, as it only scales the magnitudes of correctly classifie… view at source ↗
Figure 8
Figure 8. Figure 8: Bagging ensemble collapsed and degenerated solution proportions in individual run on COPA dataset. 21 view at source ↗
Figure 9
Figure 9. Figure 9: Accuracies of 10 random runs by Roberta-Large. FFT FocalLoss PCGrad LNSR NoisyTune Ours Methods 0.50 0.55 0.60 0.65 0.70 0.75 0.80 Accuracy (a) MultiRC. FFT FocalLoss PCGrad LNSR NoisyTune Ours Methods 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 Accuracy (b) RTE. FFT FocalLoss PCGrad LNSR NoisyTune Ours Methods 0.55 0.60 0.65 0.70 0.75 0.80 0.85 Accuracy (c) BoolQ. FFT FocalLoss PCGrad LNSR NoisyTune… view at source ↗
Figure 10
Figure 10. Figure 10: Accuracies of 10 random runs by Llama-3.2-1B. 22 view at source ↗
Figure 11
Figure 11. Figure 11: Accuracies of 10 random runs by ViT-base. 23 view at source ↗
read the original abstract

Fine-tuning pretrained models has become a standard approach to adapting pretrained knowledge to improve the accuracy on new sparse, imbalance datasets. However, issues arise when optimization falls into a collapsed state, where the model gets stuck, leading to degraded performance and unstable training. One possible reason for this is the cancellation of gradients across training examples. To address this problem, we propose a novel algorithm, dynamic scaled gradient descent (\mName), that directly modifies the gradients returned by training examples, specifically, scaling down the gradients of correctly classified examples using a dynamic scaler. This strategy offers both theoretical and empirical advantages in improving training stability. Experiments on a variety of benchmark datasets, spanning multiple tasks and large pretrained models, demonstrate that our method consistently reduces performance variance and surpasses the accuracy of existing approaches.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 2 minor

Summary. The paper claims that fine-tuning pretrained models on sparse, imbalanced classification datasets often leads to collapsed training states due to gradient cancellation across examples. It proposes Dynamic Scaled Gradient Descent (DSGD or mName), which modifies gradients by dynamically scaling down those from correctly classified examples via a dynamic scaler. The method is asserted to offer both theoretical and empirical advantages for training stability. Experiments across benchmark datasets, tasks, and large pretrained models are said to show consistently lower performance variance and higher accuracy than existing approaches.

Significance. If the central mechanism holds and the dynamic scaling specifically mitigates cancellation-induced collapse without harming hard examples or introducing new instabilities, the approach could offer a lightweight, practical modification to standard optimizers that improves reliability of fine-tuning on challenging datasets. This would be relevant for imbalanced classification tasks in vision and NLP, though its significance would depend on whether gains exceed those from generic regularization or hyperparameter tuning.

major comments (4)
  1. [Abstract, Method] Abstract and Method section: The dynamic scaler is the core of the proposed algorithm, yet no explicit formula, derivation, or analysis is supplied to justify the claimed theoretical advantages or to show how the scaling factor is computed from predictions/loss without introducing circularity or additional fitted parameters.
  2. [Experiments] Experiments section: No details are provided on baselines, hyperparameter controls, statistical significance tests, or variance metrics; the claim of 'consistently reduces performance variance and surpasses the accuracy' cannot be evaluated without these, and no ablation isolates the correctness-based scaling from generic effects like altered effective learning rates.
  3. [Method, Experiments] Method and Experiments: The assumption that gradient cancellation is the primary collapse cause and that selectively scaling correct examples fixes it is not verified; missing are gradient-norm histograms, per-example contribution analysis, or ablations (e.g., scaling by loss magnitude instead of binary correctness label). On imbalanced data this risks systematically down-weighting majority-class gradients.
  4. [Method] Method: The dynamic scaler schedule is listed among free parameters, but no guidance is given on its selection or whether it remains truly dynamic and non-circular when defined in terms of the evolving model predictions.
minor comments (2)
  1. [Abstract, Method] Notation for mName and the dynamic scaler should be introduced clearly with an equation on first use.
  2. [Introduction] Related work on gradient scaling, stable fine-tuning, and handling of imbalanced data should be expanded with specific citations.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the careful review and valuable feedback on our paper. We have addressed each of the major comments below and will make corresponding revisions to the manuscript to improve clarity, provide missing details, and strengthen the empirical validation of our claims.

read point-by-point responses

  1. Referee: Abstract and Method section: The dynamic scaler is the core of the proposed algorithm, yet no explicit formula, derivation, or analysis is supplied to justify the claimed theoretical advantages or to show how the scaling factor is computed from predictions/loss without introducing circularity or additional fitted parameters.

    Authors: We agree that an explicit formula for the dynamic scaler was not sufficiently detailed in the abstract and method sections. In the revised manuscript, we will provide the exact formula: the scaling factor for a correctly classified example is s_t = 1 - p_{y,t-1}, where p_{y,t-1} is the predicted probability of the true class from the previous iteration. This ensures the scaling is dynamic and non-circular, as it relies on prior predictions. We will include a derivation demonstrating that this reduces the norm of gradients from easy examples, thereby preventing cancellation with hard examples' gradients. No additional parameters are fitted; the schedule refers to how the threshold for 'correct' is annealed over training. revision: yes

  2. Referee: Experiments section: No details are provided on baselines, hyperparameter controls, statistical significance tests, or variance metrics; the claim of 'consistently reduces performance variance and surpasses the accuracy' cannot be evaluated without these, and no ablation isolates the correctness-based scaling from generic effects like altered effective learning rates.

    Authors: We acknowledge these omissions in the experimental reporting. The revised paper will include comprehensive details on all baselines and their hyperparameter tuning procedures, including the search spaces used. We will report mean accuracy and standard deviation over multiple random seeds (at least 5), along with statistical significance tests (e.g., t-tests) comparing our method to baselines. To isolate the effect of correctness-based scaling, we will add an ablation study comparing it to uniform scaling and loss-based scaling, showing that the specific use of correctness label is key to the improvements rather than just changing the effective learning rate. revision: yes

  3. Referee: Method and Experiments: The assumption that gradient cancellation is the primary collapse cause and that selectively scaling correct examples fixes it is not verified; missing are gradient-norm histograms, per-example contribution analysis, or ablations (e.g., scaling by loss magnitude instead of binary correctness label). On imbalanced data this risks systematically down-weighting majority-class gradients.

    Authors: This is a fair point, and we will strengthen the verification in the revision. We will add gradient-norm histograms comparing standard GD and DSGD, as well as per-example gradient contribution analysis to show reduced cancellation. An ablation on scaling by loss magnitude versus binary correctness will be included to demonstrate the advantage of our approach. Regarding imbalanced data, we will analyze per-class performance and gradient contributions to confirm that majority classes are not unduly down-weighted, as scaling depends on individual example correctness rather than class frequency. While we believe the empirical results support our assumption, these additions will provide direct evidence. revision: partial

  4. Referee: Method: The dynamic scaler schedule is listed among free parameters, but no guidance is given on its selection or whether it remains truly dynamic and non-circular when defined in terms of the evolving model predictions.

    Authors: We will add explicit guidance on selecting the dynamic scaler schedule in the revised method section. Specifically, we recommend initializing with a high scaling threshold and gradually decreasing it based on validation loss monitoring. The computation remains truly dynamic and non-circular because the scaling factor at step t is determined solely from the model's output probabilities at step t-1, prior to computing the current gradients and update. We will include pseudocode to clarify the sequence of operations. revision: yes

Circularity Check

0 steps flagged

No significant circularity; method definition does not reduce to its inputs by construction.

full rationale

The provided abstract and description introduce a dynamic scaling rule applied to gradients of correctly classified examples, with the scaler described as dynamic (i.e., state-dependent on current model outputs). No equations, parameter-fitting procedure, self-citation chain, or uniqueness theorem are exhibited that would make the claimed stability gain equivalent to the input data or to a fitted constant by definition. The correctness label is computed from the forward pass in the standard way; applying a scaling factor to those gradients is an explicit algorithmic choice rather than a renaming or tautological re-expression of the loss surface. Empirical claims rest on benchmark comparisons rather than on any self-referential prediction. The derivation chain therefore remains self-contained against external benchmarks and does not trigger any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that gradient cancellation is the dominant cause of collapse and on an unspecified dynamic scaling rule whose parameters are not shown to be derived from first principles.

free parameters (1)
  • dynamic scaler schedule
    The factor that reduces gradients of correct examples must be controlled by at least one tunable or data-dependent parameter whose value is not derived in the abstract.
axioms (1)
  • domain assumption Gradient cancellation across training examples is a primary driver of collapsed states in fine-tuning.
    The abstract states this as 'one possible reason' and builds the entire method on it.

pith-pipeline@v0.9.0 · 5421 in / 1337 out tokens · 38453 ms · 2026-05-07T05:54:58.129299+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    findings-acl.1016/

    URL https://aclanthology.org/2023. bigpicture-1.10/. Mosbach, M., Andriushchenko, M., and Klakow, D. On the stability of fine-tuning bert: Misconceptions.Explana- tions, and Strong Baselines. arXiv, 2020. Nesterov, Y .Introductory Lectures on Convex Optimiza- tion: A Basic Course. Springer Publishing Company, Incorporated, 1 edition, 2014. ISBN 1461346916...

    work page doi:10.18653/v1/2025.findings-emnlp 2023

  2. [2]

    emnlp-main.1173/

    URL https://aclanthology.org/2025. findings-emnlp.1246/. Pecher, B., Cegin, J., Belanec, R., Simko, J., Srba, I., and Bielikova, M. Fighting randomness with random- ness: Mitigating optimisation instability of fine-tuning using delayed ensemble and noisy interpolation. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.), Findings of the Association fo...

    work page doi:10.18653/v1/2024.findings-emnlp 2025

  3. [3]

    In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp

    URL https://aclanthology.org/2024. findings-emnlp.644/. Phang, J., F´evry, T., and Bowman, S. R. Sentence encoders on stilts: Supplementary training on intermediate labeled- data tasks.arXiv preprint arXiv:1811.01088, 2018. Picard, D. Torch. manual seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vi...

    work page doi:10.18653/v1/2023.acl-long 2024

  4. [4]

    AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

    URL https://aclanthology.org/2023. acl-long.877/. Wu, C., Wu, F., Qi, T., and Huang, Y . Noisytune: A little noise can help you finetune pretrained language models better. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 680–685, 2022. Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman,...

    work page internal anchor Pith review arXiv 2023