Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

The paper claims that adding an importance state to each attention memory slot turns in-context learning into a continual learner that protects old associations, and that a widely used state-space model is its fast-forgetting limit.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:07 UTC pith:BTT56JT5

load-bearing objection A credibly derived dual-state Bayesian update for gated linear attention; the mechanism claims outrun the current experiments, but the framework deserves serious engagement. the 3 major comments →

arxiv 2602.09075 v4 pith:BTT56JT5 submitted 2026-02-09 cs.LG cs.AI

Learning to Remember, Learn, and Forget in Attention-Based Models

Djohan Bonnet , Jamie Lohoff , Jan Finkbeiner , Elidona Shiqerukaj , Emre Neftci This is my paper
classification cs.LG cs.AI
keywords in-context learningcatastrophic forgettingmetaplasticitylinear attentionstate space modelsassociative memoryBayesian inferencecontinual learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Palimpsa, an attention layer that treats in-context learning as a continual learning problem: a fixed-size memory must keep old associations while absorbing new ones, without replay. The proposed mechanism is Bayesian metaplasticity—each memory slot tracks a precision (importance) value that controls how much that slot is allowed to change, and a forgetting term that releases stale knowledge. The authors claim this prevents both catastrophic forgetting and catastrophic remembering, and they show that several existing gated linear attention models are special cases of the same Bayesian update, with Mamba2 emerging as the extreme where forgetting dominates. The contribution is a single principle that both explains the behavior of existing linear-attention architectures and offers a plug-in upgrade that improves recall and reasoning at fixed inference memory cost.

Core claim

Palimpsa's central claim is that a fixed-size attention memory behaves like a true palimpsest—retaining significant writing, washing away stale writing—when each memory element carries its own uncertainty. Concretely, the layer maintains two states per slot: a precision vector I_t, which acts as the slot's importance, and the associative memory μ_t. On each token the precision is updated as I_t = α_t I_{t−1} + (1−α_t)I_prior + β_t ⊗ k_t^2, and the memory as μ_t = α_t (I_{t−1}/I_t) ⊙ μ_{t−1} + (1/I_t) ⊙ ((β_t ⊙ v_t) ⊗ k_t), where α_t is an input-dependent forgetting gate tied to an effective memory window N_t and β_t is an input gate. The paper argues that this update, derived from a variatio

What carries the argument

The carrying object is the Palimpsa update pair, Eq. (5). The first line maintains a precision state I_t, a per-slot measure of how much evidence each memory element has accumulated; the second updates the associative memory μ_t using the ratio of old to new precision. The engine behind it is the variational free energy with three terms—plasticity (new key–value evidence), stability (protection of the previous memory weighted by its precision), and forgetting (a pull toward a prior controlled by the window N_t). The forgetting gate α_t = exp(−A d_t) turns the scalar memory window into an input-dependent quantity, which is what lets the model decide in context when to remember and when to rel

Load-bearing premise

The load-bearing premise is that forgetting can be simulated by discounting all past evidence by a fixed fraction at every step—the geometric-mean proxy for the unavailable oldest data point in Appendix A.2—and if that proxy is wrong, the effective memory window and every gate derived from it inherit the error.

What would settle it

On the multi-query associative recall task, measure accuracy separately for the earliest key–value pairs in each sequence as sequence length grows. If the metaplastic model truly protects old associations, its early-pair recall should decay much more slowly than a non-metaplastic gated linear attention baseline; equal decay would refute the central stability claim. A sharper version is to implement the exact sliding-window posterior instead of the geometric-mean proxy and verify whether the Palimpsa update still matches it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any non-metaplastic gated linear attention or delta-rule model can be upgraded to the metaplastic form by adding the precision state and fine-tuning; the paper demonstrates this for both a delta-rule-style backbone and a state-space-style backbone.
  • At 760M scale, fine-tuned metaplastic variants outperform their non-metaplastic counterparts by roughly 0.6 average accuracy points on commonsense reasoning, with the largest improvement on LAMBADA, a task that requires holding broad context.
  • On the multi-query associative recall benchmark, the performance gap between metaplastic and non-metaplastic variants grows with sequence length, consistent with the model learning to slow forgetting (larger N_t) as task difficulty increases.
  • Because Mamba2 is the high-forgetting limit of Palimpsa, the framework provides a continuous dial between Mamba2-like behaviour and full metaplastic memory; a practical consequence is pre-training with fast non-metaplastic kernels followed by a short metaplastic fine-tuning phase.
  • At inference time the added cost of metaplasticity is modest: the recurrent state remains fixed-size, and the benchmarked attention kernel runs at roughly a quarter of a simple gated linear attention kernel's throughput in isolation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension: because the diagonal precision treats every memory element independently, a natural next step is to reintroduce selective correlations between key dimensions, which could strengthen associative recall but would require trading away some of the single-pass efficiency.
  • Extension: the learned effective window N_t could serve as an interpretability probe—per layer, it tells how far back a model is actually looking; the paper does not test this diagnostic use.
  • Extension: one could anneal the forgetting dial from the Mamba2 limit toward full metaplasticity during training, testing whether the benefit can be acquired gradually rather than through a separate fine-tuning phase.
  • Extension: replacing the geometric-mean stale-data proxy with an exact sliding-window likelihood would isolate how much of the empirical gain is due to the approximation itself rather than to the metaplastic update.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Palimpsa, a gated linear attention layer that casts in-context learning as online Bayesian continual learning. It derives an update rule with a per-state importance/precision variable I_t, which modulates the learning rate to protect old associations while a forgetting factor releases stale information. The authors show that several existing architectures (Longhorn, Deltanet, Gated Deltanet, MesaNet, and Mamba2) can be viewed as special cases or approximations of this framework, and they introduce a fine-tuning scheme to convert a pre-trained Mamba2 model into a metaplastic Palimpsa. Experiments on MQAR and on Commonsense Reasoning benchmarks (170M and 760M models) report consistent gains over non-metaplastic baselines, with the largest gains on long-sequence MQAR and on LAMBADA.

Significance. If the central claims hold, the paper makes a useful contribution: it provides a variational-Bayesian derivation of a practical gated linear attention update with per-state metaplasticity, unifies several existing models under one formalism, and demonstrates a concrete recipe for upgrading Mamba2 into a memory-enhanced model. The derivations in the appendices are explicit and self-contained, and the experimental protocol is careful (8 seeds, curriculum, token-budget parity for fine-tuning). The main caveat is that the headline mechanism — preventing in-context catastrophic forgetting — is not isolated by the reported experiments, and two theoretical steps (the geometric-mean forgetting proxy and the Mamba2 limit) need sharper treatment before the claims are fully persuasive.

major comments (3)
  1. [§3.1, Fig. 2, Eq. (5)] The headline claim that Palimpsa 'prevents in-context catastrophic forgetting' is supported only by average MQAR accuracy over all queries. Such an average cannot distinguish genuine preservation of early KV pairs from a recency bias produced by the forgetting factor in Eq. (5). Please report accuracy conditioned on key position (or pair age), or a retention curve. Without this, the mechanism claim is underdetermined; Fig. 3's loss and metaplasticity ratio do not resolve the issue.
  2. [Appendix A.2, Eqs. (8)–(9)] The forgetting mechanism is built on the geometric-mean proxy p(d_{t−N}|S) ≈ (q_{θ_{t−1}}(S)/p(S))^{1/N}. All subsequent claims about the effective memory window N_t and the stability-plasticity trade-off inherit this approximation. The proxy is plausible but is neither validated nor bounded. Please provide a small-scale comparison with the exact truncated posterior (e.g., in a linear regression setting) or a theoretical error bound, and state which conclusions would change if the proxy is inaccurate.
  3. [§2.3, §2.4, Table 1] The Mamba2 special case is stated in §2.3 as arising from a high forgetting rate, but I_t ≈ I_prior requires more than small α_t: the β_t ⊗ k_t^2 term in Eq. (5) must also be negligible relative to I_prior. The reparameterization in §2.4 (β→0, large-value limit) supplies this, but the notation v*_t = β_t ⊙ v_t = SiLU(θ_v x_t) is inconsistent as written (a product cannot equal the SiLU activation unless additional scaling is defined). Please clarify the limit and the notation; the phrase 'where forgetting dominates' is too strong for the actual condition.
minor comments (5)
  1. [§2.2, Eq. (5)] I_prior is introduced without specifying whether it is a scalar or a vector; Eq. (5) and Table 1 use it in ways that suggest a vector. Please define its dimensions.
  2. [Appendix A.5] The sentence 'this is a stronger approximation than a standard mean-field (diagonal covariance) assumption' is confusing. Deriving the full solution and then discarding off-diagonals is not stronger than solving under a diagonal constraint; please rephrase.
  3. [Appendix A.6] The first sentence says 'Gated Deltanet can be derived similarly by suppressing the forgetting', but the derivation that follows includes forgetting (α_t). The wording should be corrected.
  4. [§2.4] The notation v*_t appears without prior definition and is used inconsistently with v_t. Please define it clearly and state its role in the kernel.
  5. [Table 2] The column labeled 'LMB. ppl↓' appears to be followed by accuracy columns without a clear separator; check the formatting so that perplexity and accuracy are visually distinct.

Circularity Check

0 steps flagged

No significant circularity: Palimpsa's update rule is derived from a stated Bayesian free energy, and the Mamba2/Deltanet recoveries are explicit algebraic limits rather than fitted predictions.

full rationale

The derivation chain is self-contained in the relevant sense. Equation (5) is obtained by setting to zero the analytical gradients of the free energy in Eq. (3) under the explicitly stated diagonal-covariance approximation (Appendices A.3-A.5); the update is not assumed as the answer. The forgetting mechanism enters through the weighted posterior of Appendix A.2, which the paper transparently labels as a proxy ('As a proxy, one can use the geometric mean of the likelihoods...') and attributes to prior work (Bonnet et al., 2025). This is a stated modeling assumption, not a hidden fit or a renamed prediction; the paper's claims inherit it honestly. The recoveries of Deltanet and Mamba2 are also explicit: Table 1 and Section 2.3 derive Mamba2 by taking I_t approximately equal to I_prior in Eq. (5), and Appendix A.6 derives Gated Deltanet under stated diagonal/first-order simplifications. These are demonstrative mathematical reductions made after knowing the target update, so they are not falsifiable empirical predictions, but neither are they circular in the derivation sense. The MESU self-citation is load-bearing for the forgetting schedule, but it refers to a published, externally evaluated prior method rather than an unverified uniqueness claim, and the present contribution applies it to attention with a new derivation and ablations. The MQAR average-accuracy evidence is an identifiable experimental-design concern about isolating retention by key position, not a circularity in the mathematical derivation. Therefore no circular step can be exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central derivation rests on a small set of free parameters (gates, β, prior importance) and approximation choices (diagonal covariance, geometric-mean forgetting, β→0 limit). These are clearly stated but are neither forced by external benchmarks nor machine-checked.

free parameters (4)
  • Forgetting gate α_t (via A and d_t) = trained; A initialized U(0.01,0.16) on MQAR, U(0,16) on language; Δt bias log-ramp 0.001–0.1
    Controls memory window N_t = 1/(1−α_t) and is the central stability-plasticity dial in Eq. 5.
  • Input-dependent importance gain β_t = trained; Palimpsa-D uses a dedicated projection, Palimpsa-M uses SiLU(θ_v x_t)
    Determines per-state plasticity and enters both the I_t and μ_t updates; vector-valued in Palimpsa.
  • Prior importance I_prior = not stated explicitly; in the Deltanet derivation it is set to 1
    Floors the precision and sets the asymptotic Mamba2-like learning rate; its value is a design choice that influences the Mamba2 limit.
  • Per-head β amplitude scale b_scale = fine-tuning schedule in [0.1, 1.0]
    Introduced in Section 2.4 to force I_t≈I_prior at fine-tuning start, enabling the continuous Palimpsa-to-Mamba2 transition.
axioms (5)
  • domain assumption Gaussian likelihood p(v|k,β,S) with diagonal precision diag(β)
    Models self-attention output as linear regression in S; standard in linear-attention-as-optimization literature but not true of softmax attention.
  • domain assumption Weighted-posterior forgetting proxy using geometric mean of past likelihoods
    Eq. 8–9 in Appendix A.2 substitutes unavailable p(d_{t−N}|S) with (q_{θ_{t−1}}(S)/p(S))^{1/N}; load-bearing for the claimed memory horizon.
  • domain assumption Variational posterior q_{θ_{t-1}} approximates the previous weighted posterior
    Needed to derive the recursive free energy; standard assumed-density filtering / online VI.
  • ad hoc to paper Diagonal covariance approximation
    Appendix A.5 discards off-diagonal precision after deriving the full solution, enabling vector β_t and hardware-efficient scans; affects fidelity of the posterior.
  • ad hoc to paper Reparameterization v* = β⊙v = SiLU(θ_v x_t) with b_scale controlling β amplitude
    Constructed in Section 2.4 so Palimpsa-M limits to Mamba2 when β→0; this is an engineering device for the claimed special-case result.
invented entities (1)
  • Importance state I_t (per-synapse precision trace) no independent evidence
    purpose: Tracks per-state uncertainty/importance, modulates plasticity, implements metaplastic forgetting.
    An internal architectural state; its only observable effects are benchmark outcomes, so there is no evidence for it outside the paper's own experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 20508 in / 12798 out tokens · 140162 ms · 2026-08-03T03:07:35.594165+00:00 · methodology

0 comments
read the original abstract

In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks. However, in gated linear attention models, this memory has a fixed capacity and is prone to interference, especially for long sequences. We propose Palimpsa, a self-attention model that views ICL as a continual learning problem that must address a stability-plasticity dilemma. Palimpsa uses Bayesian metaplasticity, where the plasticity of each attention state is tied to an importance state grounded by a prior distribution that captures accumulated knowledge. We demonstrate that various gated linear attention models emerge as specific architecture choices and posterior approximations, and that Mamba2 is a special case of Palimpsa where forgetting dominates. This theoretical link enables the transformation of any non-metaplastic model into a metaplastic one, significantly expanding its memory capacity. Our experiments show that Palimpsa consistently outperforms baselines on the Multi-Query Associative Recall (MQAR) benchmark and on Commonsense Reasoning tasks.

Figures

Figures reproduced from arXiv: 2602.09075 by Djohan Bonnet, Elidona Shiqerukaj, Emre Neftci, Jamie Lohoff, Jan Finkbeiner.

Figure 1
Figure 1. Figure 1: Bayesian Metaplasticity Attention. Self-attention in autoregressive transformers is inherently a continual learning problem, and as such can suffer from catastrophic forgetting. Metaplasticity dynamically modifies the learning rate to preserve important prior information. (Bottom-left) Illustration of Bayesian metalearning: qθt is the (variational) distribution over memory states S at time step t. (Right) … view at source ↗
Figure 3
Figure 3. Figure 3: Palimpsa’s Learning Dynamics: Memory window Nt (blue), averaged over the context length, and the metaplasticity ratio (orange), defined on the final state importance as (Imax − Imin)/Imin, and the training loss (pink). A higher ratio indicates stronger differentiation between plastic and consolidated synapses. Shaded regions represent the standard deviation over 8 seeds. holds). The exponential growth of N… view at source ↗
Figure 2
Figure 2. Figure 2: Curriculum MQAR experiments. Accuracy averaged over 8 seeds for the best learning rate per model. Individual run ac￾curacies are shown as black dots; error bars represent ±1 standard deviation. Task difficulty increases with sequence length L. “w/o Meta” indicates that metaplasticity was disabled for those models. explained). We include Gated Deltanet as a baseline for comparison. Results are reported over… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the Palimpsa-M and Palimpsa-D architectures. While Palimpsa-M adopts the Mamba-2 configuration by relying on the attention layer for channel mixing, Palimpsa-D incorporates an explicit gated MLP. Additionally, Palimpsa-D introduces a dedicated bt parameter to decorrelate input integration from the forgetting dynamics dictated by dt. A.2. Bayesian Forgetting Following the work of (Bonnet et … view at source ↗
Figure 5
Figure 5. Figure 5: Inference Speed Benchmark. Throughput (thousands of tokens/s) comparison between Palimpsa and Simple GLA on an NVIDIA GeForce RTX 3090. Palimpsa matches the baseline’s scaling behavior while maintaining a consistent 4× factor due to the dual-state update overhead. We benchmarked the fused recurrent kernel on a single NVIDIA GeForce RTX 3090 GPU (B = 1, L = 32, H = 1), varying the model dimension D ∈ {512, … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Black-Mamba: Biologically-Inspired Leaky Accumulation for Conceptual Knowledge under Distribution Drift

    cs.AI 2026-07 conditional novelty 6.0

    Gating test-time memory writes on leaky accumulated surprisal preserves most of the adaptation benefit while roughly halving the number of updates.

  2. Memory by Design: Probabilistic Sequence Layers

    stat.ML 2026-05 unverdicted novelty 6.0

    The design-model framework unifies sub-quadratic sequence models as Bayesian filters and introduces a covariance-tracking Bayesian Layer that improves retrieval robustness beyond training regimes on MQAR and RULER benchmarks.

Reference graph

Works this paper leans on

31 extracted references · 27 linked inside Pith · cited by 2 Pith papers

  1. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    9 Learning to Remember, Learn, and Forget in Attention-Based Models Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    Pith/arXiv arXiv 2010

  2. [9]

    Is in-context learn- ing in large language models bayesian? a martingale perspective.arXiv preprint arXiv:2406.00793,

    Falck, F., Wang, Z., and Holmes, C. Is in-context learn- ing in large language models bayesian? a martingale perspective.arXiv preprint arXiv:2406.00793,

    Pith/arXiv arXiv

  3. [10]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Lineartime sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Pith/arXiv arXiv

  4. [11]

    and Goyal, N

    Hahn, M. and Goyal, N. A theory of emergent in-context learning as implicit structure induction.arXiv preprint arXiv:2303.07971,

    Pith/arXiv arXiv

  5. [12]

    Under- standing catastrophic forgetting and remembering in con- tinual learning with optimal relevance mapping.NeurIPS 2021 Workshop MetaLearn Poster

    Kaushik, P., Gain, A., Kortylewski, A., and Yuille, A. Under- standing catastrophic forgetting and remembering in con- tinual learning with optimal relevance mapping.NeurIPS 2021 Workshop MetaLearn Poster. Available on arXiv preprint arXiv:2102.11343,

    Pith/arXiv arXiv 2021

  6. [13]

    W., et al

    Kim, S., Hooper, C., Wattanawong, T., Kang, M., Yan, R., Genc, H., Dinh, G., Huang, Q., Keutzer, K., Mahoney, M. W., et al. Full stack optimization of transformer infer- ence: a survey.arXiv preprint arXiv:2302.14017,

    Pith/arXiv arXiv

  7. [14]

    Longhorn: State space models are amortized online learners.arXiv preprint arXiv:2407.14207,

    Liu, B., Wang, R., Wu, L., Feng, Y ., Stone, P., and Liu, Q. Longhorn: State space models are amortized online learners.arXiv preprint arXiv:2407.14207,

    Pith/arXiv arXiv

  8. [17]

    Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,

    Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y . Socialiqa: Commonsense reasoning about social interac- tions.arXiv preprint arXiv:1904.09728,

    Pith/arXiv arXiv 1904

  9. [19]

    Remembering transformer for continual learning.arXiv preprint arXiv:2404.07518,

    Sun, Y ., Fujisawa, I., Juliani, A., Sakuma, J., and Kanai, R. Remembering transformer for continual learning.arXiv preprint arXiv:2404.07518,

    Pith/arXiv arXiv

  10. [21]

    Transformers learn incontext by gradient descent

    von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn incontext by gradient descent. InInternational Conference on Machine Learning, pp. 3515135174. PMLR, 2023a. von Oswald, J., Niklasson, E., Schlegel, M., Kobayashi, S., Zucchet, N., Scherrer, N., Miller, N., Sandler, M., Vladym...

    Pith/arXiv arXiv

  11. [22]

    Self-expansion of pre-trained models with mixture of adapters for continual learning.arXiv preprint arXiv:2403.18886,

    Wang, H., Lu, H., Yao, L., and Gong, D. Self-expansion of pre-trained models with mixture of adapters for continual learning.arXiv preprint arXiv:2403.18886,

    Pith/arXiv arXiv

  12. [23]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    Pith/arXiv arXiv

  13. [24]

    com/fla-org/flash-linear-attention

    URL https://github. com/fla-org/flash-linear-attention. Yang, S., Wang, B., Shen, Y ., Panda, R., and Kim, Y . Gated linear attention transformers with hardwareefficient train- ing.arXiv preprint arXiv:2312.06635,

    Pith/arXiv arXiv

  14. [25]

    Gated delta net- works: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta net- works: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    Pith/arXiv arXiv

  15. [26]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Pith/arXiv arXiv 1905

  16. [27]

    Continual learning through synaptic intelligence

    Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In Precup, D. and Teh, Y . W. (eds.),Proceedings of the 34th International Con- ference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pp. 39873995. PMLR, 0611 Aug 2017a. URL https://proceedings. mlr.press/v70/zenke17a.html. Zenke, F., Poole...

    Pith/arXiv arXiv

  17. [28]

    com/fla-org/flame

    URL https://github. com/fla-org/flame. Zhang, Y ., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averag- ing, parameterization, and generalization.arXiv preprint arXiv:2305.19420,

    Pith/arXiv arXiv

  18. [29]

    Appendices As described in the main text, Palimpsa-M builds upon Mamba-2, while Palimpsa-D is based on Deltanets

    11 Learning to Remember, Learn, and Forget in Attention-Based Models A. Appendices As described in the main text, Palimpsa-M builds upon Mamba-2, while Palimpsa-D is based on Deltanets. These two models utilize distinct gated linear attention layers and separate backbones, allowing our metaplasticity ablation studies to evaluate two highly different point...

    2025

  19. [1024]

    For all models, word embeddings are tied with the language modeling head

    Optimizer AdamW Learning rate [1e-3,2.15e-3,4.64e-3,1e-2] Random seeds{1, 2, 3, 4, 5, 6, 7, 8} Language Modelling Experiments We evaluate Palimpsa across two primary model scales: 170M and 760M parameters. For all models, word embeddings are tied with the language modeling head. The decay parameters A are initialized by sampling from a uniform distributio...

    2025

  20. [1989]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031,

    Pith/arXiv arXiv

  21. [2016]

    A., Rabinowitz, N

    Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks.arXiv preprint arXiv:1606.04671,

    Pith/arXiv arXiv

  22. [2017]

    von Oswald, J., Henning, C., Sacramento, J., and Grewe, B. F. Continual learning with hypernetworks.arXiv preprint arXiv:1906.00695,

    Pith/arXiv arXiv 1906

  23. [2018]

    and Gu, A

    Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    Pith/arXiv arXiv

  24. [2019]

    Learning to (learn at test time).arXiv preprint arXiv:2310.13807,

    Sun, Y ., Li, X., Dalal, K., Hsu, C., Koyejo, S., Guestrin, C., Wang, X., Hashimoto, T., and Chen, X. Learning to (learn at test time).arXiv preprint arXiv:2310.13807,

    Pith/arXiv arXiv

  25. [2020]

    Mambacl: Optimizing selective state space model in null space for continual learning.arXiv preprint arXiv:2411.15469,

    Cheng, D., Lu, Y ., He, L., Zhang, S., Yang, X., Wang, N., and Gao, X. Mambacl: Optimizing selective state space model in null space for continual learning.arXiv preprint arXiv:2411.15469,

    Pith/arXiv arXiv

  26. [2021]

    What learning algorithm is incontext learn- ing? investigations with linear models.arXiv preprint arXiv:2211.15661,

    Aky¨urek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is incontext learn- ing? investigations with linear models.arXiv preprint arXiv:2211.15661,

    Pith/arXiv arXiv

  27. [2022]

    Arora, A., Jurafsky, D., Potts, C., and Goodman, N. D. Bayesian scaling laws for in-context learning, 2024a. URLhttps://arxiv.org/abs/2410.16531. Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Re, C. Zoology: Measuring and improving recall in efficient language models. InThe Twelfth International Conference on Learn...

    arXiv

  28. [2023]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  29. [2024]

    Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663,

    Behrouz, A., Zhong, P., and Mirrokni, V . Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663,

    Pith/arXiv arXiv

  30. [2025]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817,

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y ., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817,

    Pith/arXiv arXiv

  31. [3090]

    We benchmarked the fused recurrent kernel on a singleNVIDIA GeForce RTX 3090GPU ( B= 1, L= 32, H= 1 ), varying the model dimension D∈ {512,1024,2048}

    Palimpsa matches the baseline’s scaling behavior while maintaining a consistent 4× factor due to the dual-state update overhead. We benchmarked the fused recurrent kernel on a singleNVIDIA GeForce RTX 3090GPU ( B= 1, L= 32, H= 1 ), varying the model dimension D∈ {512,1024,2048}. As shown in Figure 5, Palimpsa demonstrates robust throughput, peaking at rou...

    2048