Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Omega-S: A Functional Resilience Index for LLM Fine-Tuning

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Omega-S, a data-free penalty on the weight matrix, claims higher code retention after prose fine-tuning than no regularisation, weight decay, and EWC on Llama-3-8B with LoRA.

desk verdict Honest, self-critical empirical paper: Omega-S gives a real retention gain on one LoRA task pair, but the missing plasticity measurement makes the central claim conditional. read the letter →

arxiv 2608.03887 v1 pith:IHZBEAIA submitted 2026-08-04 cs.LG cs.NEq-bio.MN

Alberto Acedo This is my paper
classification cs.LGcs.NEq-bio.MN
keywords catastrophicforgettingfine-tuningregularizationLoRAdegreevarianceweightgraphHumanEvalLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Omega-S, a drop-in penalty computed from the weight matrix alone, and claims it reduces catastrophic forgetting when a large language model is fine-tuned on new data. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, it reports higher retained code capability than no regularisation on 9 of 10 seeds and higher than tuned weight decay on 10 of 10 and tuned EWC on 8 of 10, all without needing previous-task data, a Fisher matrix, or a stored copy of old weights. The paper also measures that the penalty's practical effect is to reduce the variance of node degrees: three of its four topological factors are numerically inert because of a saturating logistic map, and a reformulation that revives the clustering channel makes retention worse on all ten seeds. If these claims hold, Omega-S offers a cheap, data-free regulariser for sequential fine-tuning, and its explicit mechanism measurement provides a template for checking what a regulariser actually does.

What carries the argument

The engine is the log-ratio composite penalty. For a weight matrix W, the paper forms A = σ(|W W^T|) for non-square modules and σ(|W|) for square ones, reads the degree sequence k = A1, builds the graph Laplacian L = diag(k) − A, and penalises log(M·Coex/(C·D)), where D is mean(A), Coex is variance of k, C is a Hutchinson-estimated normalised trace of A^3, and M estimates the Fiedler value λ2(L) via power iteration. The paper's direct measurement shows that, because the logistic map saturates affinities into [0.5, 1), C is pinned near its ceiling, D and M are numerically inert, and the only factor that moves is Coex. As implemented, the composite is therefore a degree-variance regulariser, e

What would settle it

Measure the model's perplexity on Wikitext-2 (the prose fine-tuning set) after training with and without Omega-S under the paper's ten-seed protocol; if Omega-S's prose perplexity is substantially worse than no regularisation, the retention gain is a plasticity cost, not a stability win.

Watch

Extended reading notes

Core claim

Sequential fine-tuning from code to prose on Llama-3-8B with LoRA, measured by HumanEval pass@1 over ten seeds, retains more code capability with Omega-S than with no regularisation on 9 of 10 seeds: absolute 0.173 to 0.238, retention ratio 62.9% to 84.1%, with one-sided sign test p=0.011 and Wilcoxon p=0.006. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), with every arm re-measured in the same session. The paper further claims, from direct elasticity and run-time measurements, that the objective reduces in practice to a penalty on the variance of node degrees: three of its four factors have elasticity at or below 1e-4 and do not move during

Load-bearing premise

Omega-S only matters if the penalty does not simply stop the model from learning the new task; the paper reports no measurement of second-task plasticity, so the retention gain could be trivial if learning is suppressed.

Editorial extensions

If this is right

  • Practitioners can add Omega-S to a LoRA training loop in three lines and get retention gains without storing old-task data, a Fisher matrix, or previous weights; per-step overhead is under 4% at K=10.
  • Because the effect is measured to act through degree-variance reduction rather than clustering, Omega-S in this formulation should be described as a degree-variance regulariser, not a working topological method.
  • A contrast-preserving variant that revives the clustering channel loses on all ten seeds, so the saturated clustering channel is not what holds the method back.
  • Seed-paired comparisons in this setting carry a repeat standard deviation of 0.104 in retention ratio; mean differences below roughly 0.066 are not distinguishable, bounding claims made on ten-seed runs.
  • The method exceeds the row-norm control on 8 of 10 seeds even when the control is tuned on its own grid, suggesting degree variance is not merely a reparametrisation of weight decay.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The untouched question is plasticity: if a Wikitext perplexity test shows Omega-S preserves prose learning, the method is a genuine continual-learning regulariser; if not, the retention gain may be a suppression artifact.
  • Since degree variance is not row-norm variance (r ≈ 0.60), Omega-S may occupy an unexplored lever in LoRA continual learning; a direct single-task generalisation test would tell whether it also helps ordinary training.
  • The measured run-to-run variation implies that published seed-paired comparisons in low-rank LLM fine-tuning should average several repetitions per cell before drawing conclusions, and that the two module-shape branches (square vs non-square) need deliberate standardisation.
  • A contrast-preserving affinity, such as a rank-normalised or temperature-scaled map, would let a future variant test whether a genuinely clustering-driven penalty can help, but the current evidence points the other way.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Omega-S, a parameter-only regularizer for fine-tuning LLMs, derived from an ecological network-resilience index. The penalty is built from the third spectral moment of a logistic-saturated weight Gram matrix and is applied in log-ratio composite form. On Llama-3-8B with LoRA, sequential code-to-prose fine-tuning, it reports that Omega-S retains more HumanEval pass@1 than no regularization on 9/10 seeds (0.173 vs 0.238 absolute; retention 62.9% vs 84.1%), and beats tuned weight decay and EWC. A direct elasticity measurement shows that, under the implemented logistic construction, only the degree-variance term is active; the clustering, density, and modularity factors are numerically inert. The paper also quantifies run-to-run variation (SD 0.104 in retention ratio), reports a failed contrast-preserving reformulation, and lists several limitations, including the fact that plasticity on the second task is not measured.

Significance. If the retention result is robust, Omega-S is a practically attractive drop-in regularizer because it requires no previous-task data, no Fisher matrix, and no stored old weights. The manuscript is unusually honest and scientifically useful: it reports negative results, quantifies seed-repeat variability, flags the selection-on-evaluation-seeds flaw, and directly measures the mechanism rather than relying on the objective's nominal form. These reporting practices are exemplary and should be credited. However, the paper's usefulness claim is conditional on the regularizer not merely suppressing learning of the second task, and this is explicitly untested. The selection protocol also weakens the head-to-head comparisons with tuned baselines. These issues are material but addressable.

major comments (4)
  1. [Section 3.2] Plasticity on the second task is unmeasured. The text states: 'Whether Omega-S costs plasticity on the second task is untested here and would require a Wikitext perplexity measurement we did not make.' Because HumanEval is the only evaluation metric and is measured after both code and prose fine-tuning, a regularizer that simply suppresses learning on Wikitext-2 would trivially preserve code ability. The retention claim itself is internally consistent, but the paper's stated value as a fine-tuning regularizer presupposes that the model still learns prose. This is a load-bearing missing control. At minimum, report Wikitext-2 perplexity (or equivalent second-task accuracy) for the no-regularizer, weight-decay, EWC, and Omega-S arms in the same ten-seed protocol; this is a feasible extension of the existing experiment.
  2. [Section B.11] Hyperparameter selection is performed on two seeds (42 and 123) that are also in the ten-seed evaluation set. The paper explicitly acknowledges this: 'the selection set is a subset of the evaluation set rather than disjoint from it.' This affects the reported tuned baselines and also Omega-S's calibration target 0.03, which lies at the edge of the swept grid. The no-regularizer arm is unaffected and remains the cleanest comparison, but the pairwise claims against tuned weight decay and EWC (10/10, 8/10) are optimistic because the same favorable selection seeds are reused in the evaluation. Please provide either a disjoint selection/evaluation split, or a sensitivity analysis showing that the qualitative conclusions are unchanged when the two selection seeds are removed from the win/loss counts.
  3. [Section 3.6] The reference implementation branches between sigma(|W W^T|) for non-square W and sigma(|W|) for square W, so the runs on Llama-3-8B use two different constructions (q_proj is square, v_proj is not). The paper states that the degree-variance term differs by a factor of roughly 400 between branches on q_proj. This means the evaluated 'Omega-S' is not one well-defined penalty but a mixed procedure. The mechanism ablation by module type partly addresses this, but the central ten-seed result and the reported elasticity measurements should be reproduced under a single, deliberately chosen construction. Otherwise the method's identity and reproducibility remain ambiguous.
  4. [Section 2.4 vs. Section B.12] There is a quantitative inconsistency in the reported cosine-composite negative result. Section 2.4 states the cosine composite loses with mean retention 0.547 against 0.841, while Appendix B.12 reports 0.537 against 0.766. These may come from different orientations of M (corrected vs. as-implemented), but the main text does not say so. Please align the numbers or explicitly state which configuration each refers to, since this affects the interpretation of the negative result.
minor comments (4)
  1. [Eq. (3)] The formula for C appears to contain a typo: 'cTr(A^3)/||A||_F^3 + ε + ε' has a duplicated epsilon term. Please correct.
  2. [Quick Start / Section 2.3] The Quick Start code adds the penalty at every optimization step, but Section 2.3 recommends applying it every K steps (K=10). The code block should include the K gating or be amended to match the description.
  3. [Table 3] The 'he_A (task-A capability)' row shows that the magnitude-only arm ends task A at 0.193 HumanEval versus 0.289 for the alignment-only arm. The paper acknowledges this, but the retention ratios in the table should include this caveat in the table caption or as a visible footnote, since comparing ratios with different denominators is easy to misread.
  4. [Abstract] The abstract states 'three lines in an existing training loop' while the actual recommended usage requires a K-step gating and Hutchinson probe configuration; this is slightly oversold. Consider saying 'a few lines' or showing the complete usage.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: retention is an external empirical measurement; self-citations motivate but do not carry the claim.

full rationale

The main claim—that Omega-S retains more code capability after code→prose fine-tuning than no regularisation, tuned weight decay, or tuned EWC—is measured against HumanEval, an external benchmark that is not an input to the penalty. The penalty coefficient is calibrated at run time to fix the penalty gradient as a fixed fraction of the task gradient (Section 2.1: 'λ_Omega calibrated at run time so that the penalty gradient is a fixed fraction of the task gradient'; Section 3.1: 'a λ_Omega calibrated at run time so that the penalty gradient is a fixed fraction 0.03 of the task gradient'), which does not target the retention outcome. The mechanism claim that the composite 'reduces in practice to a penalty on the variance of node degrees' is a direct measurement (Section 3.3) rather than an assumption, and the paper explicitly reports a contrast-preserving variant that revives the clustering channel and makes retention worse on all ten seeds. The self-citations—the soil-microbiome studies [1,2], the patents [3,4], and the companion theoretical manuscript [21]—supply the index's provenance and vocabulary, but the retention result does not depend on any unverified claim from those works; Tr(A^3)=Σλ_i^3 is standard linear algebra. The paper also flags the principal limitation: 'Whether Omega-S costs plasticity on the second task is untested here and would require a Wikitext perplexity measurement we did not make' (Section 3.2). That is a missing control for the usefulness interpretation, not a circular step, because the measured quantity (HumanEval after prose) is not defined in terms of the penalty. The selection-set overlap noted in Appendix B.11 (hyperparameters chosen on two seeds that are also in the ten-seed evaluation set) is a selection-bias concern, not an equation-level reduction. Overall, the derivation chain is not circular: the penalty is a design choice, the outcome is an external measurement, and the mechanism is empirically decomposed rather than asserted.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The method's practical contribution rests on a domain-assumption bridge from soil ecology to neural weight graphs, plus an adaptive penalty coefficient, while the paper's own measurements show the active mechanism is a simple degree-variance statistic rather than the topological index that motivates the name.

free parameters (5)
  • lambda_Omega (per-seed calibration) = 16 to 41 across seeds (calibrated so penalty gradient is 3% of task gradient)
    Penalty strength is not fixed; it is recalibrated per seed to maintain a gradient ratio, so the method has an adaptive free parameter.
  • Calibration target (penalty gradient fraction) = 0.03
    Chosen from sweep {0.03,0.1,0.3,0.5} on seeds 42 and 123, which are also in the evaluation set; 0.03 is at the edge of the grid, so the optimum may not be bracketed.
  • Application frequency K = 10 steps
    From recommended range [10,50]; affects overhead and effect.
  • Hutchinson probe count n_p = 16 in ten-seed experiment
    n_p=3 was dominated by sampling noise; 16 chosen.
  • Module subset per application = 8 sampled modules
    Only a sampled subset of eight modules receives the penalty per step; the selection procedure is not fully specified (random? which?).
assumptions (5)
  • standard math Tr(A^3) = sum lambda_i^3 and Hutchinson estimator unbiasedness
    Used to define the penalty and its efficient computation (Equations 1, 6, 7).
  • domain assumption The ecological index Omega = C*D/(M*Coex) and the directional associations with resilience from vineyard soil studies [1,2]
    Motivates the penalty direction and the M=1/lambda_2 orientation; the transfer from soil networks to neural weight graphs is assumed, not derived.
  • domain assumption The graph A = sigma(|W W^T|) of the weight matrix captures structure relevant to forgetting
    The core modeling assumption that a bounded map of the Gram matrix yields a meaningful network whose degree variance is a useful training signal.
  • domain assumption HumanEval pass@1 after a prose fine-tune measures retention of the original code capability
    The evaluation metric is treated as the target capability; the paper acknowledges it is not a plasticity measure.
  • domain assumption Seed-paired comparisons are valid despite GPU non-determinism
    The paper measures std 0.104 in retention ratio under identical configuration, showing the seed does not fully identify the run, yet paired tests are still used throughout (Section 3.4).
invented entities (1)
  • Weight monopolies
    purpose: Conceptual framing: a small subset of neurons concentrates connectivity, claimed to be the root cause of forgetting; used to motivate Omega-S.
    The paper does not directly demonstrate that 'weight monopolies' exist in Llama-3-8B or that their reduction causes retention; the measured active mechanism is reduction of node-degree variance, which is a related but distinct quantity. Also, an earlier two-regime hypothesis (topological vs norm-balancing) was refuted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Omega-S: A Functional Resilience Index for LLM Fine-Tuning." pith.science (2026). https://pith.science/paper/IHZBEAIA

@misc{pith2026260803887,
  author       = {Pith},
  title        = {Pith review of: Omega-S: A Functional Resilience Index for LLM Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHZBEAIA}},
  note         = {Machine review of arXiv:2608.03887}
}
read the original abstract

Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.

Figures

Figures reproduced from arXiv: 2608.03887 by the authors.

Figure 1
Figure 1. Per-seed retention in the ten-seed experiment, all arms tuned. Omega-S (right bars) [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 31 canonical work pages

  1. [1]

    Network Properties of Local Fungal Communities Reveal the Anthropogenic Disturbance Consequences of Farming Practices in Vineyard Soils.mSystems

    Ortiz-Álvarez R, Ortega-Arranz H, Ontiveros VJ, de Celis M, Ravarani C, Acedo A, Belda I. Network Properties of Local Fungal Communities Reveal the Anthropogenic Disturbance Consequences of Farming Practices in Vineyard Soils.mSystems. 2021;6(3):e00344-21. doi: 10.1128/mSystems.00344-21

    work page doi:10.1128/msystems.00344-21 2021

  2. [2]

    Soil Microbial Diversity and Network Organization Respond to Land Use and Agricultural Inputs Worldwide.Global Change Biology

    Saati-Santamaría Z, Pérez-Gorjón S, Abel-Schaad D, et al. Soil Microbial Diversity and Network Organization Respond to Land Use and Agricultural Inputs Worldwide.Global Change Biology. 2026;32(7):e70984. doi: 10.1111/gcb.70984

    work page doi:10.1111/gcb.70984 2026

  3. [3]

    Methods and systems for gen- erating and applying agronomic indices from microbiome-derived parameters

    Acedo A, Ortega-Arranz H, Almonacid D, Ferrero A (2022). Methods and systems for gen- erating and applying agronomic indices from microbiome-derived parameters. US Patent Application Publication No. US 2022/0268756 A1 (Appl. No. 17/665,332, filed 4 February 2022), Biome Makers Inc

    work page 2022

  4. [4]

    Methods and systems for evaluating ecological disturbance of an agricultural micro- biome based upon network properties of organism communities

    Ortiz-Alvarez R, Ferrero A, Imam N, Acedo A, Belda-Aguilar I, Almonacid D (2021). Methods and systems for evaluating ecological disturbance of an agricultural micro- biome based upon network properties of organism communities. WIPO Publication No. WO 2021/119528 A1 (PCT/US2020/064668), Biome Makers Inc

    work page 2021

  5. [5]

    Sharpness-Aware Minimization for Efficiently Improving Generalization.ICLR

    Foret P, Kleiner A, Mobahi H, Neyshabur B. Sharpness-Aware Minimization for Efficiently Improving Generalization.ICLR. 2021. 30

    work page 2021

  6. [6]

    Fiedler Regularization: Learning Neural Networks with Graph Sparsity

    Tam E, Dunson D. Fiedler Regularization: Learning Neural Networks with Graph Sparsity. ICML. 2020. arXiv:2003.00992. Extended version: Spectral Gap Regularization of Neural Networks. arXiv:2304.03096. 2023

    work page Pith review arXiv 2020

  7. [7]

    A stochastic estimator of the trace of the influence matrix for Lapla- cian smoothing splines.Communications in Statistics — Simulation and Computation

    Hutchinson MF. A stochastic estimator of the trace of the influence matrix for Lapla- cian smoothing splines.Communications in Statistics — Simulation and Computation. 1990;19(2):433–450

    work page 1990

  8. [8]

    Decoupled Weight Decay Regularization.ICLR

    Loshchilov I, Hutter F. Decoupled Weight Decay Regularization.ICLR. 2019

    work page 2019

Show all 42 references
  1. [9]

    LoRA: Low-Rank Adaptation of Large Language Models.ICLR

    Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. LoRA: Low-Rank Adaptation of Large Language Models.ICLR. 2022

    2022

  2. [10]

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.VLDB

    Zhao Y, Gu A, Varma R, Luo L, Huang CC, Xu M, et al. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.VLDB. 2023

    2023

  3. [11]

    Model selection and estimation in regression with grouped variables.Journal of the Royal Statistical Society: Series B

    Yuan M, Lin Y. Model selection and estimation in regression with grouped variables.Journal of the Royal Statistical Society: Series B. 2006;68(1):49–67

    2006

  4. [12]

    Emergence of scaling in random networks.Science

    Barabási AL, Albert R. Emergence of scaling in random networks.Science. 1999;286(5439):509–512

    1999

  5. [13]

    Hutch++: Optimal Stochastic Trace Estima- tion.SIAM Symposium on Simplicity in Algorithms

    Meyer RA, Musco C, Musco C, Woodruff DP. Hutch++: Optimal Stochastic Trace Estima- tion.SIAM Symposium on Simplicity in Algorithms. 2021

    2021

  6. [14]

    Yin L, Wu J, Zhang Z, et al. (2024). Outlier Weighted Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.arXiv:2310.05175

    2024 arXiv

  7. [15]

    The Llama 3 herd of models.arXiv:2407.21783

    Llama Team, AI @ Meta (2024). The Llama 3 herd of models.arXiv:2407.21783

    2024 arXiv

  8. [16]

    Chen M, Tworek J, Jun H, Yuan Q, Pinto HPO, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374

    2021 arXiv

  9. [17]

    Code Alpaca: an instruction-following LLaMA model for code gener- ation.https://github.com/sahil280114/codealpaca

    Chaudhary S (2023). Code Alpaca: an instruction-following LLaMA model for code gener- ation.https://github.com/sahil280114/codealpaca

    2023

  10. [18]

    Pointer sentinel mixture models.Interna- tional Conference on Learning Representations

    Merity S, Xiong C, Bradbury J, Socher R (2017). Pointer sentinel mixture models.Interna- tional Conference on Learning Representations

    2017

  11. [19]

    LLM.int8(): 8-bit matrix mul- tiplication for transformers at scale.Advances in Neural Information Processing Systems 35:30318–30332

    Dettmers T, Lewis M, Belkada Y, Zettlemoyer L (2022). LLM.int8(): 8-bit matrix mul- tiplication for transformers at scale.Advances in Neural Information Processing Systems 35:30318–30332

    2022

  12. [20]

    SmoothQuant: accurate and efficient post-training quantization for large language models.International Conference on Machine Learning, PMLR 202:38087–38099

    Xiao G, Lin J, Seznec M, Wu H, Demouth J, Han S (2023). SmoothQuant: accurate and efficient post-training quantization for large language models.International Conference on Machine Learning, PMLR 202:38087–38099

    2023

  13. [21]

    Acedo A. (2026). The Functional Symbiotic Resilience Index: topological entropy, Wasser- stein curvature bounds and non-equilibrium thermodynamics of complex networks. A frame- work linking network structure, non-equilibrium thermodynamics and discrete curvature. Preprint

    2026

  14. [22]

    Muon- OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

    Lu B, Deng Z, Zhang R, Hu B, Zhao Y, Tian Y, Mou C, Lin G, Li X (2026). Muon- OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning. arXiv:2605.08949

    2026 arXiv

  15. [23]

    Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures.arXiv:2601.03889

    Delibasoglu I (2026). Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures.arXiv:2601.03889. 31

    2026

  16. [24]

    HidekelIN,RavivD(2026).CatastrophicForgettingisLow-Rank: AFunction-SpaceTheory for Continual Adaptation.arXiv:2606.18024

    2026 arXiv

  17. [25]

    Non-determinism in TensorFlow ResNets.arXiv:2001.11396

    Morin M, Willetts M (2020). Non-determinism in TensorFlow ResNets.arXiv:2001.11396

    2020 arXiv

  18. [26]

    Deterministic Implementations for Reproducibility in Deep Reinforcement Learning.arXiv:1809.05676

    Nagarajan P, Warnell G, Stone P (2018). Deterministic Implementations for Reproducibility in Deep Reinforcement Learning.arXiv:1809.05676

    2018 arXiv

  19. [27]

    Pinto A F, et al. (2021). On the reproducibility of neural network predictions. arXiv:2105.05482

    2021 arXiv

  20. [28]

    Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adap- tation.arXiv:2603.02224

    Steele B (2026). Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adap- tation.arXiv:2603.02224

    2026

  21. [29]

    From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning.arXiv:2606.26629

    Ning E, Xue W, Lou D, Guo Y (2026). From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning.arXiv:2606.26629

    2026 arXiv

  22. [30]

    Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning.arXiv:2601.18699

    Laitinen-Fredriksson Lundstrom-Imanov GOY (2026). Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning.arXiv:2601.18699

    2026

  23. [31]

    Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabska-Barwinska A, et al. (2017). Overcoming catastrophic forgetting in neural networks.PNAS114(13):3521–3526

    2017

  24. [32]

    Learning without forgetting.IEEE TPAMI40(12):2935–2947

    Li Z, Hoiem D (2017). Learning without forgetting.IEEE TPAMI40(12):2935–2947

    2017

  25. [33]

    Orthogonal gradient descent for continual learning.AISTATS, PMLR 108:3762–3773

    Farajtabar M, Azizan N, Mott A, Li A (2020). Orthogonal gradient descent for continual learning.AISTATS, PMLR 108:3762–3773

    2020

  26. [34]

    Orthogonal subspace learning for language model continual learning.Findings of EMNLP 2023, 10658–10671

    Wang X, Chen T, Ge Q, Xia H, Bao R, Zheng R, Zhang Q, Gui T, Huang X (2023). Orthogonal subspace learning for language model continual learning.Findings of EMNLP 2023, 10658–10671

    2023

  27. [35]

    Nayak NS, Killamsetty K, Han L, Bhandwaldar A, Chanda P, Xu K, Wang H, Pareja A, Silkin O, Eyceoz M, et al. (2025). Sculpting subspaces: Constrained full fine-tuning in LLMs for continual learning.arXiv:2504.07097

    2025 arXiv

  28. [36]

    Wang X, Zhang Y, Chen T, Gao S, Jin S, Yang X, Xi Z, Zheng R, Zou Y, Gui T, et al. (2023). TRACE: A comprehensive benchmark for continual learning in large language models.arXiv:2310.06762

    2023 arXiv

  29. [37]

    Lu Y, et al. (2024). AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models.arXiv:2410.10912

    2024 arXiv

  30. [38]

    Anonymous. (2025). Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models.arXiv:2505.01731

    2025 arXiv

  31. [39]

    Tang S, Sieberling O, Kurtic E, Shen Z, Alistarh D. (2025). DarwinLM: Evolutionary Struc- tured Pruning of Large Language Models.arXiv:2502.07780

    2025 arXiv

  32. [40]

    Mocanu D C, Mocanu E, Nguyen P H, Gibescu M, Liotta A. (2016). A Topological Insight into Restricted Boltzmann Machines.Machine Learning, 104, 243–270

    2016

  33. [41]

    Sun M, Liu Z, Bair A, Kolter J Z. (2024). A Simple and Effective Pruning Approach for Large Language Models (Wanda).ICLR 2024

    2024

  34. [42]

    Frantar E, Alistarh D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One Shot.ICML 2023. 32 Correspondence: Alberto Acedo (Biome Makers Inc.). The author declares no competing financial interests beyond the pending patent application referenced herein....

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.