Pith. sign in

REVIEW 3 major objections 4 minor 66 references

Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adam's squared-gradient accumulator, recycled as the 'Squisher', performs as well as the expensive Fisher diagonal across merging, pruning, masking, task embedding, and continual learning.

desk verdict Clean derivation and a genuinely useful free-lunch idea, but the 'comparable performance' claim is shakier than the paper lets on—no error bars, and its own appendix shows the comparison flips with checkpoints. read the letter →

arxiv 2507.18807 v1 pith:CYKS52TH submitted 2025-07-24 cs.LG

YuXin Li , Felix Dangel , Derek Tam , Colin Raffel This is my paper · ORCID
classification cs.LG
keywords FisherinformationmatrixsquaredgradientaccumulatorparameterimportanceAdamoptimizermodelmergingnetworkpruningelasticweightconsolidationtaskembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adaptive optimizers like Adam already compute a quantity during training — an exponential moving average of squared gradients — that the paper argues can be recycled as a free approximation of the diagonal of the Fisher Information Matrix, the standard measure of parameter sensitivity used in model merging, pruning, sparse fine-tuning, task embedding, and continual learning. Computing the Fisher diagonal after training is expensive and needs the training data, because it requires per-example gradients on sampled labels; the recycled accumulator, which the paper calls the Squisher, costs nothing beyond loading optimizer state. The paper first pins down what the accumulator actually is — a moving-average version of the diagonal of the joint empirical Fisher, off by a known scaling factor — and then tests the substitution in six settings across five applications. Replacing the Fisher with the Squisher changes performance only slightly, in both directions, while both clearly beat the Fisher-free baselines in every setting. If the finding holds, Fisher-based methods become available for any model trained with an adaptive optimizer, with no extra computation and no need for training data or gradient code.

What carries the argument

The load-bearing object is the squared gradient accumulator $v^{(t)} = \alpha v^{(t-1)} + (1-\alpha)\big(\tfrac{1}{N}\sum_n g_n^{(t)}\big)^2$, the exponential moving average of squared mean mini-batch gradients that Adam and its derivatives maintain for learning-rate adaptation. The key identity is structural: the accumulator squares a summed (mean) gradient rather than summing squared per-example gradients, so it is exactly a moving-average version of the diagonal of the joint empirical Fisher information matrix, whose entry is $N\big(\tfrac{1}{N}\sum_n g_n\big)^2$; the joint Fisher and the standard Fisher coincide (Lin et al., 2024), which lets the paper say the accumulator is a Fisher diagonal of a recognizable kind, differing only in that the average runs over the training trajectory and in the scaling factor $N$. This identity does the argumentative work: it converts the question 'can we reuse the accumulator?' from an analogy into the specific empirical question of whether trajectory-averaging and scaling break the usefulness of an otherwise standard Fisher, which the experiments then answer.

What would settle it

Use the paper's own recipe: train VGG-13 on CIFAR-100 with Adam at $\beta_2 = 0.95$, or stop training early, then prune 75% of parameters using the Squisher ranking and compare accuracy with Fisher pruning and random pruning; the paper's ablation predicts a sharp drop in the $\beta_2 = 0.95$ regime, and a configuration where the Squisher's pruned accuracy falls to the random-mask level would falsify the drop-in claim. A second, more direct check is to compute the rank correlation between the Squisher's per-parameter scores and the label-sampled Fisher diagonal on the same checkpoint and see whether a low-correlation regime exists in which any application's performance gap to the Fisher-free baseline disappears.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the squared gradient accumulator $v^{(t)} = \alpha v^{(t-1)} + (1-\alpha)\big(\tfrac{1}{N}\sum_n g_n^{(t)}\big)^2$, the second-moment estimate Adam keeps for adaptive learning rates, is a workable drop-in replacement for the diagonal of the empirical Fisher, $F = \sum_n g_n^2$, in applications that use the Fisher as a parameter-importance score. The substitution is not a loose metaphor: the accumulator squares the average gradient over a batch (sum-then-square), which makes it a trajectory-averaged version of the diagonal of the joint empirical Fisher, $N\big(\tfrac{1}{N}\sum_n g_n\big)^2$, identified with the standard Fisher through the joint/standard equivalence; the two differences are the moving average over training instead of an evaluation at the final parameters, and the factor $N$ in scale. In the experiments, the two methods stay close — the Squisher slightly ahead in Fisher merging of eight fine-tuned T5 variants and in the in-class task-embedding rankings, slightly behind in pruning of VGG-13, and nearly identical in uncertainty-based gradient matching, FISH Mask, and EWC — while both clearly beat the Fisher-free baselines in every setting; in EWC, the one application where rescaling matters, multiplying the accumulator by the dataset size restores comparable behaviour. Computing the Fisher in these settings took between 2.5 seconds and roughly 14 hours, while the Squisher costs under a tenth of a second to load.

Load-bearing premise

The load-bearing assumption is that the exponential moving average of squared gradients, accumulated with gradients taken at the moving parameter values throughout training and with the default smoothing coefficient, remains a faithful proxy for the Fisher diagonal evaluated at the final parameters; the paper offers no theoretical argument for this correspondence, only experiments, and its own ablations show the proxy degrades when training is cut short or when $\beta_2$ is lowered.

Editorial extensions

If this is right

  • Any model trained with Adam, AdamW, or a similar adaptive optimizer already carries a parameter-importance score in its optimizer state, so Fisher-based techniques — merging, pruning, sparse fine-tuning, task embeddings, and EWC — can run with zero extra computation and no second pass over the data.
  • Fisher-based methods become usable precisely where they were previously impractical: when a released model comes without its training data, or when per-example gradient code is unavailable, the accumulator alone suffices.
  • The practical caveats are bounded and concrete: the Squisher needs a fully trained model and the default smoothing coefficient, since lowering $\beta_2$ to 0.95 cut EWC accuracy on split CIFAR-100 from 75.30 to 61.53, and merging from under-trained checkpoints ('best model' selection) makes the Squisher lose its edge.
  • Applications that are invariant to rescaling can use the accumulator as-is, and EWC only needs the accumulator multiplied by the dataset size $N$, so switching from the Fisher adds no tuning burden beyond the $\lambda$ search EWC requires anyway.
  • Because the optimizer state now has value on its own, the paper argues that sharing optimizer statistics alongside released models would let anyone apply Fisher-based analysis to any checkpoint for free.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The training-length sensitivity suggests the Squisher is effectively a Fisher diagonal averaged over a neighbourhood of parameter values near the end of training; the natural experiment the paper does not run is to compare the Squisher against an explicit average of per-checkpoint Fisher diagonals over the final stretch of training, which would separate the contribution of the moving average from
  • A cheap hybrid is worth testing: because the Squisher costs nothing, one could compute the true Fisher on a small sample and combine the two importance rankings, potentially reducing the merging and pruning instability the paper attributes to noise without paying the full Fisher cost.
  • The overall equivalence supports a deflationary reading of the Fisher diagonal in these applications — that what carries the value is per-parameter gradient magnitude rather than the likelihood-sampling definition — which would make any optimizer statistic that tracks per-parameter gradient scale, momentum included, a candidate zero-cost importance score, a claim that is testable in the same exper
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes using the squared gradient accumulator maintained by adaptive optimizers such as Adam/AdamW as a zero-cost proxy for the diagonal Fisher information matrix, calling this proxy the Squisher. Section 2 derives an algebraic connection between the Squisher and the diagonal of the joint empirical Fisher (Eq. 12), building on the joint Fisher construction of Lin et al. (2024). Section 3 evaluates the Squisher in six settings: Fisher merging, UBGM merging, Fisher pruning, FISH masking, task embeddings, and EWC continual learning, reporting that the Squisher performs comparably to the Fisher and outperforms Fisher-free baselines. Section 4 ablates the effect of rescaling, the EMA coefficient, and the joint-Fisher construction, and Appendix A.2 examines sensitivity to training duration and checkpoint selection. The paper claims that the Squisher is a practical drop-in replacement for the Fisher in parameter-importance applications at no additional computational cost.

Significance. If the central claim holds, the paper offers a genuinely practical contribution: it removes the need for per-sample gradient computations and training-data access when computing Fisher diagonals for importance-based applications. The derivation in Section 2.3 is clean and parameter-free, and the empirical coverage across five distinct application families is a strength. The runtime measurements in Section 3.8 and Appendix A.3 quantify the savings concretely. However, the significance is tempered by the statistical fragility of the headline claim: most comparisons are single runs, and the paper's own Appendix A.2 documents checkpoint and epoch sensitivities that reverse the relative ordering of the two methods in some settings. The EWC application also relies on an empirically chosen scaling whose derivation is not supplied. These issues are addressable, but they currently limit the strength of the conclusions that can be drawn.

major comments (3)
  1. [Section 3.7, Tables 3–8, Appendix A.2] The headline claim that 'replacing the Fisher with the Squisher had little impact on performance' is not statistically supported by the reported evidence. Tables 3, 4, 6, 7, and 8 present single-run results without error bars or repeated-seed statistics; Tables 5 and 10 give five-run averages, and Table 10 shows that changing the number of training epochs flips which method is better. The per-dataset gaps are large in places (e.g., ROPES 12.6 vs 36.1 in Table 3; all-class MRR 0.312 vs 0.261 in Table 7), which is inconsistent with a characterization of 'little impact.' Appendix A.2 itself concludes that 'we cannot definitively conclude which method is superior, as the results exhibit considerable variability.' The paper should either provide confidence intervals over multiple seeds for all settings or explicitly downgrade the conclusion to 'comparable within observed variability,' with a discussion of when the differences are practically consequential.
  2. [Section 2.3, Eq. (12); Section 3.6] The scaling of the Squisher in the EWC experiment is inconsistent with the derivation in Section 2.3. The text states that when using mini-batches, one should replace N by the batch size B in all expressions. For the Adam accumulator, whose squared-gradient term is (1/B) times the sum of per-example gradients squared, the quantity in Eq. (12) becomes B times the squared mean gradient, i.e., B(1/B sum g)^2 = (1/B)(sum g)^2. Section 3.6, however, reports that scaling the Squisher by the dataset size N 'provided best performance' and admits the lack of a formal theoretical justification. This is not merely a presentation issue, because EWC is the one setting in which the scale of the importance weights changes the learning behavior. Please clarify the notation (N vs B) and either derive the correct mini-batch scaling or explicitly present the N-scaling as an empirical tuning heuristic with a sensitivity analysis.
  3. [Appendix A.2, Tables 9–10] The checkpoint and training-duration sensitivity documented in Appendix A.2 is a load-bearing limitation for the generality of the Squisher. Table 9 shows that switching from final to best checkpoints reverses the relative performance of Fisher merging (final: 53.6 vs 58.0 in favor of the Squisher; best: 59.0 vs 55.1 in favor of Fisher). Table 10 similarly shows that pruning at 15 versus 10 epochs changes which method is better. These results indicate that the Squisher's quality is not anchored to final-parameter Fisher statistics, but depends on the training trajectory and checkpointing protocol. The paper's conclusion should incorporate this dependence rather than presenting the Squisher as a universally safe drop-in replacement.
minor comments (4)
  1. [Table 2 (ablation)] The text states that even with the low beta2 value of 0.95, 'the Squisher continued to outperform the baseline,' but for split CIFAR-100 the reported accuracy is 61.53, which is below the baseline of 61.82 in Table 8. Please correct the statement or the table.
  2. [Table 6] The table header says 'a sparsity of 0.50%' but the setup in Section 3.4 describes masking 50% of the model's weights. Please reconcile the percentage.
  3. [Section 3.5 and Table 7] The claim that 'the Squisher-based task embedding produced a better mean reciprocal rank than the Fisher-based one' is only true within class groups; for all-class MRR, the Fisher is better for both classification/regression (0.312 vs 0.261) and question-answering (0.350 vs 0.301). Please qualify the claim accordingly.
  4. [Figure 1] The box labeled 'Squisher Nv(t)' does not define N; as the scaling discussion in Section 2.3 and Section 3.6 uses both dataset size and batch size, the figure should specify which quantity N denotes to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Squisher is derived by algebraic identity from the joint empirical Fisher, and its validity as a Fisher proxy rests on external empirical comparisons, not on its own construction.

full rationale

The paper's derivation chain is self-contained algebra. Equation (8) defines v(t) as an EMA of squared mini-batch mean gradients, and Equation (12) rewrites the rescaled joint empirical Fisher diagonal as N(1/N * sum g_n)^2, which is exactly the quantity whose EMA v(t) accumulates. This is an identity, not a prediction; the paper does not define the Fisher in terms of the Squisher or fit the Squisher to Fisher outputs. The theoretical bridge Fjoint = Fstd is cited from Lin et al. (2024), whose author list overlaps with the present paper, but this is a parameter-free mathematical result with stated assumptions (factorizing likelihood, zero expected score), and it is not the empirical claim being tested. The subsequent claim that the Squisher performs 'similarly' to the Fisher is established by benchmarking both against held-out tasks and Fisher-free baselines in Sections 3.1-3.6, and the paper's own Appendix A.2 quantifies instability across checkpoints and epochs rather than deriving the equivalence. EWC's lambda is tuned for both Fisher and Squisher, so no fitted parameter is renamed as a prediction. No constructed reduction, fitted-input-as-prediction, or author-imported uniqueness theorem is present; the paper's limitations (variability, beta2 sensitivity, dependence on full training) are empirical caveats, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The Squisher is not a new entity but a re-labeling of the existing squared gradient accumulator. The central derivation depends on the joint Fisher equivalence from Lin et al. (2024) and on the negative-log-likelihood interpretation of common losses. The free parameters listed are the EWC scaling heuristic (and its lambda), the optimizer's beta2, and the training duration, all of which affect the quality of the approximation.

free parameters (3)
  • EWC regularization strength lambda = grid-searched around lambda_Squisher = N * lambda_Fisher
    The EWC result requires rescaling the Squisher by N and then tuning lambda; the paper states this is a heuristic without formal justification (Section 3.6).
  • Exponential moving average coefficient beta2 = default 0.999, ablation at 0.95
    Performance of the Squisher depends on beta2 (Table 2); it is not fitted for the main method but the dependence is shown.
  • Training duration (epochs) before extracting the accumulator = varies by setting, e.g., 15 vs 10 epochs for pruning
    Appendix A.2 shows Squisher quality depends on training length; this is a setting-dependent free choice that affects results.
assumptions (3)
  • standard math Joint Fisher equals standard Fisher: F_joint = F_std (Lin et al., 2024).
    Used to justify that the sum-then-square structure of the accumulator corresponds to a valid Fisher variant. Cited from Lin et al. (2024), which includes coauthor Dangel.
  • standard math Mini-batch joint Fisher scaled by N/B is an unbiased estimator of the full joint Fisher when labels are sampled from the model likelihood.
    Used to justify using batch gradients in the Squisher derivation (Section 2.3, citing Lin et al., 2024).
  • domain assumption The loss function in Eq. (1) can be treated as a negative log likelihood per Eq. (2).
    Standard assumption for classification and regression losses, invoked to define per-sample gradients as Fisher scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator." pith.science (2026). https://pith.science/paper/CYKS52TH

@misc{pith2026250718807,
  author       = {Pith},
  title        = {Pith review of: Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYKS52TH}},
  note         = {Machine review of arXiv:2507.18807}
}
read the original abstract

The diagonal of a model's Fisher Information Matrix (the "Fisher diagonal") has frequently been used as a way to measure parameter sensitivity. Typically, the Fisher diagonal is estimated via squared sampled gradients of the model's likelihood with respect to its parameters, averaged over a few hundred or thousand examples -- a process which incurs nontrivial computational costs. At the same time, adaptive gradient methods like the ubiquitous Adam optimizer compute a moving average of the squared gradient over the course of training. This paper therefore explores whether an approximation of the Fisher diagonal can be obtained "for free" by recycling the squared gradient accumulator that has already been computed over the course of training. Through a comprehensive set of experiments covering five applications of the Fisher diagonal, we demonstrate that the "Squisher" (SQUared gradient accumulator as an approximation of the FISHER) consistently performs similarly to the Fisher diagonal while outperforming baseline methods. Additionally, we clarify the exact differences between the Squisher and the Fisher diagonal and provide empirical quantification of their respective impact.

Figures

Figures reproduced from arXiv: 2507.18807 by the authors.

Figure 1
Figure 1. The arrows represent various approximations of the Fisher Information Matrix. The central idea of the paper is highlighted through the recycling symbol. Namely, we show that squared gradient accumulators can be used to approximate the Fisher diagonal (details in Section 2). Terms in blue stem from using a loss function with mean reduction ( 1/NLsum), and should be omitted when using sum reduction (Lsum). The per-sam… view at source ↗
Figure 2
Figure 2. Performance of the Fisher, our proposed Squisher (i.e. using the squared gradient accumulator in place of the Fisher), and an applicable Fisher-free baseline across all of the settings we consider. Across all settings, the Squisher performs comparably to the Fisher and outperforms the Fisher-free baseline. tor at the end of training. Setup We exactly replicate the setup of Daheim et al. (2024) and consider the exper… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 39 canonical work pages

  1. [1]

    C., Soatto, S., and Perona, P

    Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., Soatto, S., and Perona, P. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 6430--6439, 2019

    2019

  2. [2]

    Natural gradient works efficiently in learning

    Amari, S.-I. Natural gradient works efficiently in learning. Neural Computation, 10 0 (2): 0 251--276, 1998

    work page 1998

  3. [3]

    Daheim, N., M \"o llenhoff, T., Ponti, E., Gurevych, I., and Khan, M. E. Model merging by uncertainty-based gradient matching. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  4. [4]

    B ack PACK : Packing more into backprop

    Dangel, F., Kunstner, F., and Hennig, P. B ack PACK : Packing more into backprop. In International Conference on Learning Representations (ICLR), 2020

    work page 2020

  5. [5]

    N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y

    Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in Neural Information Processing Systems, 27, 2014

    work page 2014

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019

    work page 2019

  7. [7]

    Incorporating nesterov momentum into adam, 2016

    Dozat, T. Incorporating nesterov momentum into adam, 2016. ICLR Workshop

    work page 2016

  8. [8]

    Fisher, R. A. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222 0 (594-604): 0 309--368, 1922

    work page 1922

Show all 66 references
  1. [9]

    Arcee's mergekit: A toolkit for merging large language models

    Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257, 2024

    2024 arXiv

  2. [10]

    M., Zhang, Y., Belilovsky, E., Wolf, G., and Hosseini, M

    Gomes, D. M., Zhang, Y., Belilovsky, E., Wolf, G., and Hosseini, M. S. Adafisher: Adaptive second order optimization via fisher information. arXiv preprint arXiv:2405.16397, 2024

    2024 arXiv

  3. [11]

    and Stork, D

    Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems (NIPS), 1992

    1992

  4. [12]

    J., Han, D., Yun, S., Kim, G., Uh, Y., and Ha, J.-W

    Heo, B., Chun, S., Oh, S. J., Han, D., Yun, S., Kim, G., Uh, Y., and Ha, J.-W. Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights, 2021

    2021

  5. [13]

    Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information

    Hwang, D. Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information. arXiv preprint arXiv:2405.12807, 2024

    2024 arXiv

  6. [14]

    T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  7. [15]

    and Osawa, K

    Karakida, R. and Osawa, K. Understanding approximate fisher information for fast convergence of natural gradient descent in wide neural networks. Advances in neural information processing systems, 33: 0 10891--10901, 2020

    2020

  8. [16]

    Universal statistics of fisher information in deep neural networks: Mean field approach

    Karakida, R., Akaho, S., and Amari, S.-i. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.\ 1032--1041. PMLR, 2019

    2019

  9. [17]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  10. [18]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  11. [19]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. 2009

    2009

  12. [20]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  13. [21]

    Limitations of the empirical fisher approximation for natural gradient descent

    Kunstner, F., Hennig, P., and Balles, L. Limitations of the empirical fisher approximation for natural gradient descent. Advances in Neural Information Processing Systems, 32, 2019

    2019

  14. [22]

    Optimal brain damage

    LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  15. [23]

    B., and M \"u ller, K.-R

    LeCun, Y., Bottou, L., Orr, G. B., and M \"u ller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade, pp.\ 9--50. Springer, 2002

    2002

  16. [24]

    The power of scale for parameter-efficient prompt tuning

    Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  17. [25]

    E., and Makhzani, A

    Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, R. E., and Makhzani, A. Can we remove the square-root in adaptive gradient methods? a second-order perspective. In International Conference on Machine Learning (ICML), 2024

    2024

  18. [26]

    On the variance of the adaptive learning rate and beyond

    Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020

    2020

  19. [27]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  20. [28]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  21. [29]

    S., Trivedi, P., Hougen, C., Dick, R

    Lubana, E. S., Trivedi, P., Hougen, C., Dick, R. P., and Hero, A. O. Orthoreg: Robust network pruning using orthonormality regularization. arXiv preprint arXiv:2009.05014, 2020

    2009 arXiv

  22. [30]

    P., and Wagenmakers, E.-J

    Ly, A., Marsman, M., Verhagen, J., Grasman, R. P., and Wagenmakers, E.-J. A tutorial on fisher information. Journal of Mathematical Psychology, 80: 0 40--55, 2017

    2017

  23. [31]

    and Yarats, D

    Ma, J. and Yarats, D. Quasi-hyperbolic momentum and adam for deep learning. In International Conference on Learning Representations, 2019

    2019

  24. [32]

    MacKay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003

    2003

  25. [33]

    New insights and perspectives on the natural gradient method, 2020

    Martens, J. New insights and perspectives on the natural gradient method, 2020

    2020

  26. [34]

    Matena, M. S. and Raffel, C. A. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems, 2022

    2022

  27. [35]

    McMahan, H. B. and Streeter, M. Adaptive bound optimization for online learning and stochastic optimization. In International Conference on Machine Learning (ICML), 2012

    2012

  28. [36]

    Asdl: A unified interface for gradient preconditioning in pytorch, 2023

    Osawa, K., Ishikawa, S., Yokota, R., Li, S., and Hoefler, T. Asdl: A unified interface for gradient preconditioning in pytorch, 2023

    2023

  29. [37]

    Phang, J., F \'e vry, T., and Bowman, S. R. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018

    2018 arXiv

  30. [38]

    M., Zhang, X., Pang, R

    Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., and Bowman, S. R. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint arXiv:2005.00628, 2020

    2005 arXiv

  31. [39]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 0 (1), 2020

    2020

  32. [40]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018

    2018

  33. [41]

    and Monro, S

    Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, 1951

    1951

  34. [42]

    Control chart tests based on geometric moving averages

    Roberts, S. Control chart tests based on geometric moving averages. Technometrics, 42 0 (1), 1959

    1959

  35. [43]

    and Eaton, E

    Ruvolo, P. and Eaton, E. ELLA : An efficient lifelong learning algorithm. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp.\ 507--515, Atlanta, Georgia, US...

    2013

  36. [44]

    Ill-conditioning in neural network training problems

    Saarinen, S., Bramley, R., and Cybenko, G. Ill-conditioning in neural network training problems. SIAM Journal on Scientific Computing, 14 0 (3), 1993

    1993

  37. [45]

    M., Schneider, F., and Hennig, P

    Schmidt, R. M., Schneider, F., and Hennig, P. Descending through a crowded valley-benchmarking deep learning optimizers. In International Conference on Machine Learning, 2021

    2021

  38. [46]

    M., Bazan, C., Yokota, R., Gurevych, I., Cremers, D., Khan, M

    Shen, Y., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Bazan, C., Yokota, R., Gurevych, I., Cremers, D., Khan, M. E., et al. Variational learning is effective for large deep networks. arXiv preprint arXiv:2402.17641, 2024

    2024 arXiv

  39. [47]

    K., Kim, J., and Kim, J

    Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay, 2017

    2017

  40. [48]

    Very deep convolutional networks for large-scale image recognition

    Simonyan, K. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  41. [49]

    Singh, S. P. and Jaggi, M. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33, 2020

    2020

  42. [50]

    Training neural networks with fixed sparse masks

    Sung, Y.-L., Nair, V., and Raffel, C. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  43. [51]

    Merging by matching models in task parameter subspaces

    Tam, D., Bansal, M., and Raffel, C. Merging by matching models in task parameter subspaces. Transactions on Machine Learning Research, 2024

    2024

  44. [52]

    Faster gaze prediction with dense networks and fisher pruning

    Theis, L., Korshunova, I., Tejani, A., and Husz \'a r, F. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.05787, 2018

    2018 arXiv

  45. [53]

    Thomas, V., Pedregosa, F., van Merri\"enboer, B., Manzagol, P.-A., Bengio, Y., and Roux, N. L. On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020

    2020

  46. [54]

    and Hinton, G

    Tieleman, T. and Hinton, G. Lecture 6.5—rmsprop: Divide the gradient by a running average of its recent magnitude, 2012. Coursera: Neural Networks for Machine Learning

    2012

  47. [55]

    Weight averaging for neural networks and local resampling schemes

    Utans, J. Weight averaging for neural networks and local resampling schemes. In AAAI-96 Workshop on Integrating Multiple Learned Models, 1996

    1996

  48. [56]

    van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning, 2019

    2019

  49. [57]

    M., Tice, D

    Voorhees, E. M., Tice, D. M., et al. The trec-8 question answering track evaluation. In TREC, volume 1999, pp.\ 82, 1999

    1999

  50. [58]

    Exploring and predicting transferability across nlp tasks

    Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., and Iyyer, M. Exploring and predicting transferability across nlp tasks. arXiv preprint arXiv:2005.00770, 2020

    2005 arXiv

  51. [59]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018

    2018

  52. [60]

    Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A

    Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Internation...

    2022

  53. [61]

    Ties-merging: Resolving interference when merging models

    Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  54. [62]

    Large batch optimization for deep learning: Training bert in 76 minutes

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020

    2020

  55. [63]

    Zeiler, M. D. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012

    2012 arXiv

  56. [64]

    Continual learning through synaptic intelligence, 2017

    Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence, 2017

    2017

  57. [65]

    Not all tasks are born equal: Understanding zero-shot generalization

    Zhou, J., Lin, Z., Zheng, Y., Li, J., and Yang, Z. Not all tasks are born equal: Understanding zero-shot generalization. The Eleventh International Conference on Learning Representations, 2022

    2022

  58. [66]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.