Pith. sign in

REVIEW 3 major objections 6 minor 40 references

The paper claims that a network's plasticity is the worst-case redundancy of its one-gradient-step reachable model family, and proves that a single backward pass on random-label synthetic data lower-bounds it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:14 UTC pith:Z6PSQ6MG

load-bearing objection A genuinely new information-theoretic lower bound, honestly disclosed limits, but the experiments validate the gradient-norm proxy, not local redundancy. the 3 major comments →

arxiv 2607.13432 v1 pith:Z6PSQ6MG submitted 2026-07-15 cs.LG

Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization

Jiaxuan Cheng This is my paper
classification cs.LG MSC 68T0794A17
keywords plasticitylocal redundancyuniversal compressionShtarkov sumgradient normcontinual learningcheckpoint selectionmemorization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Neural networks that train for a long time often lose the ability to pick up new tasks, but existing plasticity measures are heuristics without a shared principle. The paper puts plasticity on an information-theoretic footing by defining local redundancy: the worst-case redundancy of the small family of parameters reachable from the current weights by a single gradient step. It proves that this quantity is lower-bounded by the expected squared gradient norm on synthetic inputs whose labels are drawn from the model's own predictive distribution, making the estimate computable in a single backward pass without downstream labels. In continual image classification and time series transfer experiments, this estimate correlates with future task accuracy better than effective rank, dormant-neuron ratio, and weight norm, and it stays discriminative after validation loss plateaus. The paper is explicit that the bound is a proxy, not the true value, so the comparison is only as sound as the bound's tightness across checkpoints.

Core claim

The central claim is that plasticity is the local redundancy R**n(theta0, epsilon), defined as the worst-case redundancy of the local model family Theta(theta0, epsilon) = {theta0 - eta grad L(theta0; x^n, y^n) : eta in [0, epsilon]} — the distributions reachable by one gradient step on any dataset. Because worst-case redundancy equals the log Shtarkov sum, and because the MLE within this local family is, to first order, the exact gradient step, the paper proves Theorem 3.4: for small epsilon, R**n(theta0, epsilon) >= epsilon * E_{y^n ~ P_theta0} [ || grad_theta L(theta0; x^n, y^n) ||^2 ] + O(epsilon^2). Sampling targets from the model's own predictive distribution makes the entropy terms ca

What carries the argument

The Shtarkov sum — the sum over all possible label sequences of the best-fitting likelihood — is the object whose logarithm equals worst-case redundancy. Local redundancy feeds this sum with the gradient-defined local family instead of the whole parameter space. The load-bearing step is Proposition 3.3, which identifies the local maximum-likelihood parameter with the gradient step theta0 - epsilon grad L, so a Taylor expansion converts the memorization gain into epsilon times the expected squared gradient norm. The entropy cancellation from sampling labels y^n ~ P_theta0 is what removes the model's own output entropy and makes the gradient norm a genuine lower bound rather than a generic gra

Load-bearing premise

The load-bearing premise is that the gradient-norm lower bound is nearly equally loose at every checkpoint being compared; the paper's Limitations section states the proxy is not calibrated to the true value and could mislead where the bound is loose.

What would settle it

On a small two-layer network and a small label space, compute the exact worst-case redundancy (the log Shtarkov sum) for the local family at several checkpoints and compare its ranks with the squared-gradient-norm proxy; weak rank correlation would show the proxy fails as an estimator. Alternatively, intervene by adding a regularizer that inflates probe gradient norms without changing the true local KL-radius: if future task accuracy does not track the induced change, the metric is correlational rather than causal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Plasticity can be monitored during training with one backward pass per probe batch, with no access to future-task labels.
  • Checkpoint selection by local redundancy remains informative after validation loss plateaus, because the probe gradient norm keeps evolving while pretraining loss is flat.
  • The same estimator transfers to regression via a rate-distortion argument, so time-series models can be probed with Gaussian targets.
  • A checkpoint with higher local redundancy adapts faster during fine-tuning even when it starts from a worse initial loss, consistently with the stability-plasticity tradeoff.
  • Because the proxy is not calibrated to true redundancy, cross-checkpoint rankings are reliable only insofar as the lower bound's slack does not vary between the checkpoints being compared — the paper's own stated limitation (Section 5).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the bound's slack is roughly constant along a training run, the proxy could be used as a training signal: a regularizer that keeps the randomized-target gradient norm from shrinking is a plausible plasticity-preserving intervention, and it could be tested with existing continual-learning methods.
  • The paper reports 1–2 bits per parameter memorized on its synthetic probes, well short of a 16-bit ceiling; input distributions that saturate capacity more fully should tighten the bound and might sharpen checkpoint rankings further.
  • If the geometric reading is right — plasticity as the KL-radius of locally reachable distributions — then the Euclidean gradient norm is only one coordinate system for the radius, and whitened or natural-gradient probes should give a scale-invariant alternative; the paper does not test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces 'local redundancy' as an information-theoretic measure of plasticity. The measure is defined as the worst-case redundancy (Shtarkov sum) of a local model family: parameters reachable from the current point by one gradient step of size at most epsilon on any dataset. The main theoretical result (Theorem 3.4) proves that this local redundancy is lower-bounded by epsilon times the expected squared gradient norm on synthetic data with labels sampled from the model's predictive distribution. Since exact redundancy is intractable, the paper uses this lower bound as a computable proxy. Experiments on Continual ImageNet and time-series transfer learning show that the proxy correlates with future task accuracy and can select pretrained checkpoints that fine-tune better than checkpoints chosen by lowest validation loss. The paper also reports memorization capacity experiments (1-2 bits per parameter) to argue that the lower bound is non-vacuous.

Significance. If the lower bound is sufficiently tight, the paper provides a novel, theoretically grounded, and computationally cheap plasticity metric. The proof of Theorem 3.2 is a correct application of Jensen's inequality to the Shtarkov sum, and the entropy cancellation in Theorem 3.4 is elegant. The paper does not fit constants to data, so the reported correlations are genuine predictions. However, the empirical evaluation tests the proxy (the expected squared gradient norm) rather than the true local redundancy; the paper itself concedes in Section 5 that the proxy is uncalibrated and could mislead. The significance of the work therefore depends on resolving the tightness question, which is not addressed in the manuscript.

major comments (3)
  1. [Section 3.4, Theorem 3.4, Section 4] The load-bearing issue is the gap between the lower bound and the true local redundancy. Theorem 3.4 gives R^{**}(theta0,epsilon) >= epsilon E||grad L||^2 + O(epsilon^2), a one-sided inequality. For the empirical claims in Tables 1-3 to be about local redundancy, ranking by the proxy must agree with ranking by R^{**}; the paper provides no control on the slack, and the Limitations (Section 5) concede that 'this proxy is not calibrated to the true value and could mislead where the bound is loose.' Section 4.3 demonstrates that synthetic data can be memorized when the network is trained on it, but this does not bound the gap at evaluation checkpoints where the network has not seen the synthetic data. Thus Tables 1-3 are consistent with the conclusion that the squared gradient norm on randomized labels is a useful heuristic; they do not validate the information-theoretic quantity. The autho
  2. [Section 3.4, Proposition 3.3] The proof of Theorem 3.4 assumes a C^2 Taylor expansion of the loss along the gradient direction. Neural networks with ReLU activations (used in the vision experiments) are piecewise linear and not twice differentiable; the O(epsilon^2) expansion in Eqs. (9)-(11) is not justified at points where a ReLU unit changes status. No smoothness assumptions are stated. The authors should either restrict the theory to smooth activations, provide a subgradient-based argument that covers non-smooth cases (e.g., using directional derivatives and bounding the error uniformly), or explicitly discuss the non-smooth case. Without this, the lower bound is not rigorously established for the architectures evaluated.
  3. [Section 3.5 (Proposition 3.5, Corollary 3.6)] The regression extension is only sketched. Proposition 3.5 is justified by a rate-distortion argument with the phrase 'discretizing targets and applying Theorem 3.2 in the limit of fine quantization,' which does not constitute a proof. The relationship between worst-case redundancy and the rate-distortion function is not established. Since the time-series experiments rely on Corollary 3.6, the theoretical foundation for those experiments is incomplete. A full derivation or a precise statement with assumptions is needed.
minor comments (6)
  1. [Section 3.6] The estimator used in experiments is the mean squared gradient norm, but epsilon never appears. The paper should state the implicit normalization (e.g., epsilon=1) or explain that ranking is invariant to a positive multiplicative constant.
  2. [Table 1] The residualization procedure (regressing out task number) is not described; specify whether linear regression was used and report the distribution of residuals.
  3. [Section 4.1] The setup says 3000 tasks, but Figure 3 shows only 200 tasks. Clarify whether the correlation in Table 1 is computed on all 3000 tasks or a subset.
  4. [Section 4.2] The claim that local redundancy is 'the only plasticity metric with this property' is based on comparison with dormant ratio and weight norm only. Tables 1-2 also include distance from initialization and training gradient norm; these should be included in the checkpoint-selection experiment or the claim should be qualified.
  5. [Section 4.3] The comparison to Allen-Zhu & Li (2024) concerns knowledge capacity scaling laws, not memorization capacity; the relevance to the memorization lower bound is unclear. Also, Figure 6 caption says MobileNetV3-Small, while the text uses MobileNetV3-Large; correct the inconsistency.
  6. [Notation] The paper uses 'local redundancy' to refer both to R^{**}(theta0,epsilon) and to the proxy E||grad L||^2. Introduce separate notation (e.g., LR for the proxy) to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity: the lower-bound derivation is explicit and the proxy gap is acknowledged, not masked.

full rationale

The paper's derivation chain is self-contained. Local redundancy R**n(θ0, ε) is defined via the Shtarkov sum over a local gradient-parameter family (Def. 3.1, Eq. 4); Theorem 3.2 follows by Jensen's inequality; Prop. 3.3 is a Taylor expansion; Theorem 3.4 combines these to obtain the lower bound ε·E||∇θL||² + O(ε²). No parameter is fitted to downstream performance, and no result is imported from the author's own prior work (there are no self-citations in the reference list). The experiments estimate the lower-bound proxy, not the true local redundancy, and the paper explicitly concedes this: Section 5 states 'this proxy is not calibrated to the true value and could mislead where the bound is loose.' That is an honest validity limitation and an interpretation gap—the empirical correlations could in principle be driven by the gradient-norm heuristic alone—but it is not a definitional reduction or a fit-labeled-as-prediction step. The theoretical result has independent content and is not equivalent to its inputs by construction. Therefore no circular step meets the evidentiary bar; score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No invented physical entities are introduced; local redundancy is a defined quantity, not a postulated mechanism. The central claim rests on: (1) standard Shtarkov/Jensen machinery (sound), (2) an unproven equality in Prop 3.3 of which only the inequality direction is needed, (3) a C^2 smoothness assumption on non-smooth ReLU networks, (4) a one-sentence rate-distortion derivation for the regression extension, and (5) the load-bearing assumption, conceded in the Limitations, that the uncalibrated gradient-norm lower bound is tight enough to rank checkpoints by true local redundancy. No numerical constants are fitted to make the predicted correlations work; the free parameters are definitional (epsilon), unreported (sigma^2), and design choices (probe distribution, probe size) — with the probe distribution tuned against a future-task oracle in A.4.

free parameters (4)
  • epsilon (local step-size bound)
    Defines the local family in Def 3.1; Theorem 3.4's bound is epsilon * E||grad L||^2 + O(epsilon^2) for 'sufficiently small epsilon'. The estimator used in all experiments is the raw expected squared gradient norm — no epsilon, no remainder control — so the theorem's quantitative regime (small epsilon) never governs the experiments.
  • sigma^2 (Gaussian predictive variance for regression probe) = not reported
    Corollary 3.6 and A.2 sample regression targets y ~ N(f_theta0(x), sigma^2 I); sigma^2 is never reported for the time-series estimator, though the entropy-cancellation in the regression analogue requires the sampling distribution to be the model's predictive Gaussian.
  • synthetic probe input distribution (design choice) = vision: 5 rectangles + 5 circles, random colors/positions/sizes on random RGB background; regression: x ~ N(0, I)
    Chosen by hand and tuned in Appendix A.4 to track the 'reference' estimate on 'real, unseen images from the next task' (Pearson 0.964 vs 0.847 for Gaussian noise) — an oracle the paper itself calls inadmissible. The probe distribution controls the tightness of the lower bound, i.e., the quality of the proxy.
  • probe batch size n = 5000 (vision), 1024 (time series)
    Chosen by hand; the estimator is a Monte Carlo average over the probe batch, so n controls estimator variance and bound tightness. No sensitivity analysis over n is reported.
axioms (6)
  • standard math Minimax redundancy equals the log Shtarkov sum: R**n(Theta) = sup_{x^n} log sum_{y^n} sup_theta P_theta(y^n|x^n)
    Eq (1)-(2); attributed to Shtarkov (1987) and Davisson (1973). Backbone of Theorem 3.2.
  • standard math The change-of-measure/Jensen step with Q = P_theta0 lower-bounds log Z(x^n)
    Used in the proof of Theorem 3.2 (Eq 8). Mathematically valid; the entropy cancellation in Theorem 3.4 depends on exactly this choice of Q.
  • domain assumption The loss admits a second-order Taylor expansion with uniform O(epsilon^2) remainder over the local family
    Invoked in Prop 3.3 and Theorem 3.4. ReLU networks are not C^2 and the uniform-in-(x^n,y^n) remainder control is not established; standard in the literature but unstated here.
  • domain assumption The MLE over the local family is exactly the own-gradient step (Prop 3.3 equality)
    The local family is a union of segments along ALL datasets' gradients, so another dataset's gradient could in principle give a larger first-order loss decrease. Only the >= inequality is needed for Theorem 3.4; the stated equality is unproven.
  • domain assumption Gaussian rate-distortion implies the redundancy lower bound for regression (Prop 3.5)
    Section 3.5: 'This follows from the Gaussian rate-distortion theorem... in the limit of fine quantization.' The derivation is a sketch; the measure-theoretic bridge from the discrete-label Theorem 3.2 to continuous targets is not given.
  • domain assumption The lower bound is tight enough across checkpoints to rank true local redundancy
    The load-bearing premise for Tables 1-3. Conceded in the Limitations: the proxy 'is not calibrated to the true value and could mislead where the bound is loose', and synthetic data saturates only 1-2 bits/parameter vs a 16-bit ceiling (Sections 4.3, 5).

pith-pipeline@v1.3.0-alltime-deepseek · 13559 in / 39321 out tokens · 366244 ms · 2026-08-02T05:14:49.568087+00:00 · methodology

0 comments
read the original abstract

Plasticity -- a neural network's ability to adapt to new tasks -- is critical for continual and transfer learning. Existing measures, such as effective rank, dead neuron fraction, and weight norm, lack theoretical grounding and correlate poorly with performance on new tasks. We introduce local redundancy, an information-theoretic measure derived from universal compression theory. We define local redundancy as the worst-case redundancy of a local model family -- parameters in an infinitesimal neighborhood along gradient directions -- and show this is a principled measure of plasticity. Although local redundancy is intractable to compute exactly, we prove that the expected squared gradient norm on a synthetic memorization task provides an efficiently computable lower bound. Experiments on continual image classification and time series transfer learning demonstrate that local redundancy predicts downstream performance better than existing measures and enables pretraining checkpoint selection where validation loss plateaus.

Figures

Figures reproduced from arXiv: 2607.13432 by Jiaxuan Cheng.

Figure 1
Figure 1. Figure 1: Local redundancy: a principled, computable measure of plasticity. (a) Redundancy measures the information radius of a model class: a plastic network can reach many distributions (large radius); a rigid network cannot (small radius). (b) The expected squared gradient norm on synthetic memorization data lower-bounds local redundancy (Theorem 3.4), requiring only a single backward pass to compute. (c) Local r… view at source ↗
Figure 2
Figure 2. Figure 2: Synthetic memorization datasets. (a) Vision classifica￾tion uses input images consisting of randomly generated overlap￾ping shapes on random backgrounds, paired with labels sampled from the model’s predictive distribution. (b) Time series regression uses i.i.d. Gaussian inputs and targets with no temporal structure, sampled from the model’s predictive distribution. By Theorem 3.2, any choice of inputs x n … view at source ↗
Figure 3
Figure 3. Figure 3: Local redundancy (red) and test accuracy (blue) over 200 sequential binary classification tasks on Continual ImageNet. Each task consists of distinguishing a randomly selected pair of ImageNet classes, with training proceeding sequentially without re￾initialization. Local redundancy decreases as the network commits to learned mappings, tracking the expected loss of plasticity [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 4
Figure 4. Figure 4: Training loss (blue), validation loss (orange), and local redundancy (green) during pretraining on ETTm1 (shading indi￾cates standard deviation across seeds). Validation loss saturates after 3-6 epochs, yet local redundancy continues to increase up to 12 epochs, peaking at different epochs for different seeds. The network’s capacity to adapt continues to evolve after validation performance saturates: selec… view at source ↗
Figure 5
Figure 5. Figure 5: Fine-tuning validation loss on ETTh2 for checkpoints selected by lowest pretraining validation loss (blue) versus maxi￾mum local redundancy (orange). The high-redundancy checkpoint begins with higher loss but adapts more quickly, achieving a lower final loss on average (shading indicates standard deviation across seeds). Dashed line shows the final validation loss when training from scratch. loss as given … view at source ↗
Figure 7
Figure 7. Figure 7: Local-redundancy estimate (mean squared gradient norm) over the Continual ImageNet task sequence, computed on three input types: synthetic shapes, per-pixel Gaussian noise, and real, unseen next-task images (reference). The shape-based estimate tracks the real-image reference closely, whereas per-pixel Gaussian noise is noisier and erratic (e.g., the spike near task 150). 13 [PITH_FULL_IMAGE:figures/full_… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 6 linked inside Pith

  1. [1]

    Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents (CoLLAs), volume 232 of PMLR, 2023

    2023

  2. [2]

    Critical learning periods in deep networks

    Achille, A., Rovere, M., and Soatto, S. Critical learning periods in deep networks. In International Conference on Learning Representations (ICLR), 2019

    2019

  3. [3]

    and Bengio, Y

    Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR), Workshop Track, 2017

    2017

  4. [4]

    and Li, Y

    Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    and Adams, R

    Ash, J. and Adams, R. P. On warm-starting neural network training. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020

    2020

  6. [6]

    R., Rissanen, J., and Yu, B

    Barron, A. R., Rissanen, J., and Yu, B. The minimum description length principle in coding and modeling. IEEE Transactions on Information Theory, 44 0 (6): 0 2743--2760, 1998

    1998

  7. [7]

    Bartlett, P. L. and Mendelson, S. Rademacher and G aussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3: 0 463--482, 2002

    2002

  8. [8]

    Dynamical versus B ayesian phase transitions in a toy model of superposition

    Chen, Z., Lau, E., Mendel, J., Wei, S., and Murfet, D. Dynamical versus B ayesian phase transitions in a toy model of superposition. arXiv preprint arXiv:2310.06301, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Cover, T. M. and Thomas, J. A. Elements of Information Theory. Wiley-Interscience, 2nd edition, 2006

    2006

  10. [10]

    Davisson, L. D. Universal noiseless coding. IEEE Transactions on Information Theory, 19 0 (6): 0 783--795, 1973

    1973

  11. [11]

    ImageNet : A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet : A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 248--255, 2009

    2009

  12. [12]

    F., Lan, Q., Rahman, P., Mahmood, A

    Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632: 0 768--774, 2024

    2024

  13. [13]

    Exactly tight information-theoretic generalization bounds via binary J ensen- S hannon divergence

    Dong, Y., Guo, H., Gong, T., Wen, W., and Li, C. Exactly tight information-theoretic generalization bounds via binary J ensen- S hannon divergence. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of PMLR, pp.\ 14040--14060, 2025

    2025

  14. [14]

    K., Roy, D

    Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020

    2020

  15. [15]

    and Opper, M

    Haussler, D. and Opper, M. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Statistics, 25 0 (6): 0 2451--2492, 1997

    1997

  16. [16]

    Loss landscape degeneracy and stagewise development in transformers

    Hoogland, J., Wang, G., Farrugia-Roberts, M., Carroll, L., Wei, S., and Murfet, D. Loss landscape degeneracy and stagewise development in transformers. arXiv preprint arXiv:2402.02364, 2024. To appear in Transactions on Machine Learning Research

    Pith/arXiv arXiv 2024

  17. [17]

    V., and Adam, H

    Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q. V., and Adam, H. Searching for MobileNetV3 . In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 1314--1324, 2019

    2019

  18. [18]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 0 (13): 0 3521--3526, 2017

    2017

  19. [19]

    Implicit under-parameterization inhibits data-efficient deep reinforcement learning

    Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2021

    2021

  20. [20]

    Maintaining plasticity in continual learning via regenerative regularization

    Kumar, S., Marklund, H., and Van Roy, B. Maintaining plasticity in continual learning via regenerative regularization. arXiv preprint arXiv:2308.11958, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    S., and Solla, S

    LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS), volume 2, pp.\ 598--605, 1989

    1989

  22. [22]

    A., Pascanu, R., and Dabney, W

    Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023

    2023

  23. [23]

    New insights and perspectives on the natural gradient method

    Martens, J. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  24. [24]

    and Feder, M

    Merhav, N. and Feder, M. Universal prediction. IEEE Transactions on Information Theory, 44 0 (6): 0 2124--2147, 1998

    1998

  25. [25]

    and Wei, S

    Munn, M. and Wei, S. A B ayesian model selection criterion for selecting pretraining checkpoints. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  26. [26]

    H., Sinthong, P., and Kalagnanam, J

    Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations (ICLR), 2023

    2023

  27. [27]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of PMLR, pp.\ 16828--16847, 2022

    2022

  28. [28]

    and Wu, Y

    Polyanskiy, Y. and Wu, Y. Information Theory: From Coding to Learning. Cambridge University Press, 2025

    2025

  29. [29]

    Universal coding, information, prediction, and estimation

    Rissanen, J. Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory, 30 0 (4): 0 629--636, 1984

    1984

  30. [30]

    Fisher information and stochastic complexity

    Rissanen, J. Fisher information and stochastic complexity. IEEE Transactions on Information Theory, 42 0 (1): 0 40--47, 1996

    1996

  31. [31]

    P., and Wayne, G

    Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., and Wayne, G. Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019

    2019

  32. [32]

    A., Rabinowitz, N

    Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016

    Pith/arXiv arXiv 2016

  33. [33]

    Shtarkov, Y. M. Universal sequential coding of single messages. Problems of Information Transmission, 23 0 (3): 0 175--186, 1987

    1987

  34. [34]

    S., and Evci, U

    Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of PMLR, 2023

    2023

  35. [35]

    and Zakynthinou, L

    Steinke, T. and Zakynthinou, L. Reasoning about generalization via conditional mutual information. In Proceedings of the Thirty Third Conference on Learning Theory (COLT), volume 125 of PMLR, pp.\ 3437--3452, 2020

    2020

  36. [36]

    Vapnik, V. N. and Chervonenkis, A. Y. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16 0 (2): 0 264--280, 1971

    1971

  37. [37]

    Algebraic Geometry and Statistical Learning Theory

    Watanabe, S. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009

    2009

  38. [38]

    and Raginsky, M

    Xu, A. and Raginsky, M. Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    2017

  39. [39]

    Are all layers created equal? arXiv preprint arXiv:1902.01996, 2019

    Zhang, C., Bengio, S., and Singer, Y. Are all layers created equal? arXiv preprint arXiv:1902.01996, 2019

    Pith/arXiv arXiv 1902

  40. [40]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 11106--11115, 2021

    2021