{"id":"e89d170e-eb94-47ab-bc9d-460b964f31ce","arxiv_id":"2412.00776","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Mamba can be meta-learned as an efficient continual learner, matching or surpassing Transformers on meta-continual learning tasks with a fixed-size state and a selectivity regularizer.","lead":"This paper tests whether Mamba, an attention-free state space model, can serve as a meta-learned continual learner that processes a stream of examples through a fixed-size hidden state. It reports that Mamba matches or beats Transformers on several meta-continual learning benchmarks while using fewer parameters and faster inference, aided by a new selectivity regularizer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The selectivity regularizer's association proxy omits the A_t recurrence in Eq. (3), so Mamba may be trained to align a surrogate pattern that its hidden state does not actually implement; since Appendix C.1 shows the regularizer is necessary for convergence, this is load-bearing.","rationale":"The strongest_claim is broad: Mamba matches or exceeds Transformers on MCL benchmarks, uses a fixed-size state, is faster, and generalizes better to longer streams, domain shifts, and noise. I read the paper in good faith and find the empirical package substantial: multiple benchmark families, ablations of λ/state size/architecture, loss curves showing ℓ_slct is needed, and generalization plots. I see no fatal arithmetic error in the benchmark pipeline, and the efficiency numbers are plausible. The weakest point is the selectivity regularizer, because it is both necessary and mechanistically under-specified. Appendix C.1 reports that without ℓ_slct Mamba's loss oscillates and does not converge, so every reported Mamba result is produced with this auxiliary loss. The derivation of q_Mamba = [C_t B_j^T] identifies C and B with Q and K in linear attention, but the selective SSM recurrence in Eq. (3) also contains A_t; the unrolled output weight for past token j is C_t (∏ A_l) B_j. Unless A_l ≈ I, the surrogate q is not the true retrieval pattern. The paper does not report A_t statistics, does not ablate the proxy, and the visualizations in Appendix D use the same unvalidated surrogate. This is an internal gap between Eq. (3) and the regularization target, not a disagreement with external consensus. Because of this gap, the central claim should remain conditional: the empirical comparisons may be reproducible, but the claimed role of 'selectivity' and the generalization behavior need to be tied to the actual recurrence before the results are taken at face value. I agree with the reader's weakest_assumption, and my proposed check would settle it by comparing the surrogate and the A-corrected scores and by ablating A at inference. If the A-corrected scores are as aligned with p as the surrogate, or if clamping A does not change behavior, the concern would be retired and the paper could move toward acceptance.","tokens_in":21461,"tokens_out":11238,"duration_ms":116505,"concrete_test":"Instrument a trained MambaCL (or a small-scale reproduction; ideally with released code) and, on held-out meta-test episodes, compute for each query t and past j both s_{tj} = C_t B_j^T and e_{tj} = C_t (∏_{l=j+1}^{t} A_l) B_j. Compare the correlation of s and e with the ground-truth same-class indicator p, and the top-k agreement of the two score vectors. Then run two ablations: (i) clamp all discretized A_t to identity at inference and measure accuracy; (ii) retrain MambaCL with ℓ_slct computed on e_{tj} instead of s_{tj}. If e matches p much worse than s, if clamping A_t changes accuracy substantially, or if A-aware regularization changes the reported tables or generalization curves, the regularizer's assumed mechanism is not what drives MambaCL.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix C.1 and Fig. 6 show that without ℓ_slct, Mamba does not converge; the regularizer is therefore load-bearing. But ℓ_slct is built on q_{2t+1} = [C_{2t+1}B_j^T]_j, presented as Mamba's association pattern. Unrolling Eq. (3), the actual contribution of token j to the output at t is C_t (A_t A_{t-1} ... A_{j+1}) B_j z_j, with the input-dependent A_l terms intervening. The surrogate C_t B_j^T is exact only if A_l = I, which contradicts the selectivity/gating role of A in Mamba and is never checked. Thus the regularizer may teach the network to align C_t·B_j^T with ground-truth same-class indicators while real hidden-state retrieval is attenuated or reshaped by the A products, especially over long streams. The mechanistic claim that Mamba's selectivity enables MCL, and the length/domain/noise generalization results, rest on this unvalidated proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MambaCL, a meta-continual-learning method that trains a selective state-space model (Mamba) as a sequence-prediction continual learner with a fixed-size hidden state. To stabilize meta-training, the authors introduce a selectivity regularizer that encourages Mamba's C_t B_j^T scores to match ground-truth same-class association patterns, drawing an analogy between Mamba and linear/softmax attention. Extensive experiments on general image classification, fine-grained recognition, domain shift, regression, and length/noise generalization settings compare Mamba with Transformers, Linear Transformers, Performer, and OML, reporting that Mamba matches or exceeds Transformer performance at lower parameter count and higher inference speed. The paper also includes ablations on regularization strength, state size, architecture variants, and learning rate.","tokens_in":21715,"tokens_out":9958,"duration_ms":96070,"significance":"If the empirical claims hold, the paper is a useful demonstration that attention-free SSMs with fixed-size hidden states can be competitive with Transformers in meta-continual learning, and the length/domain/noise generalization studies are valuable for the MCL community. The paper's strengths include extensive tables with means and standard deviations, several ablation families, and the explicit convergence comparison with and without the regularizer in Fig. 6. However, the load-bearing selectivity regularizer is under-specified as a loss, and its Mamba association proxy ignores the A_t recurrence; these issues make the method description and the mechanistic interpretation currently unsupported, so the significance is conditional on resolving them.","major_comments":[{"comment":"The selectivity regularizer is not a well-defined loss as written. The paper defines ℓ_slct((x,y)) = KL(p_idx, q*_idx), where p is an unnormalized indicator vector in {0,1}^{2t} and q is, for Transformers, an unnormalized softmax score vector (the text explicitly says normalization is omitted) and, for Mamba, the raw real-valued inner-product vector [C_{2t+1}B_j^T]. KL divergence is undefined for such inputs, so the reader cannot tell what objective is actually minimized. This is not a cosmetic issue because Fig. 6 shows that Mamba does not converge without ℓ_slct, making the regularizer load-bearing. Please specify the exact computation: the normalization/softmax procedure over which positions, the handling of zero entries in p, which Mamba layer(s) provide C and B, and whether the same convention is used for the Transformer, Linear Transformer, and Performer baselines.","section":"§3.2.2 and Appendix C.1 (Fig. 6)"},{"comment":"The association proxy qMamba = [C_{2t+1}B_j^T] omits the A_t factors in the state recurrence. Unrolling Eq. (3), the contribution of token z_j to the output at step 2t+1 is C_{2t+1}(A_{2t+1}A_{2t}...A_{j+1})B_j z_j, so C_{2t+1}B_j^T equals the true retrieval weight only if all A_l are identities, which contradicts the selectivity/gating role of A in Mamba and the paper's own description in §3.1. Since Appendix C.1 makes the regularizer necessary for convergence, the method may be training Mamba to align a surrogate association pattern that its hidden state does not actually implement; the visualizations in Appendix D inherit this issue. I request a validation of the proxy (for example, comparing C_tB_j^T with the A-inclusive influence or with the actual gradient contribution on a trained model, or ablating against an A-aware weighting). Without such validation, the mechanistic interpretation and the generalization claims tied to Mamba's selectivity are not established.","section":"§3.2.2 and Eq. (3)"},{"comment":"The main results are not clearly tied to a specific Mamba architecture version. In Table 6, the row labeled Mamba-1 reports 59.7±0.5 on Cifar-100 and 90.1±0.3 on ImageNet-1K, while the row labeled Mamba-2 reports 67.1±0.4 and 93.6±0.2; the 'Mamba' entries in Table 2 (67.1±0.4 on Cifar-100 and 93.6±0.2 on ImageNet-1K) match the Mamba-2 row exactly. The body text, however, describes a Mamba block with 1-D convolution and Table 8 lists a convolution kernel size of 4, which reads like Mamba-1, and the version used for the main tables is never stated explicitly. Please state which architecture all reported results correspond to and adjust the method description and configuration table accordingly.","section":"§4.3, Table 6, and Tables 1–5"}],"minor_comments":[{"comment":"The indexing in the definition of p_{2t+1} is inconsistent: the token at step 2t+1 is x_{t+1}, so its label is y_{t+1}, not y_{2t+1}; please fix the notation.","section":"§3.2.2"},{"comment":"The captions of Figs. 10–15 describe the plots as showing Mamba's 'associations,' but the quantities plotted are the C_tB_j^T surrogate whose validity is questioned in Major Comment 2; the captions should acknowledge this.","section":"Appendix D"},{"comment":"The row label 'Delta Convolution 4' is unclear; if this denotes the convolution kernel size or dilation used in the Mamba block, please state the terminology precisely.","section":"Table 8"},{"comment":"The tables indicate that best and second-best results are highlighted in red and blue, but the text-only rendering does not show these colors; please ensure the final formatted version is unambiguous.","section":"§4.1 and Tables 1–5"}],"recommendation":"major_revision","confidential_remarks":"The empirical comparison is potentially valuable and the paper does not appear circular; my concern is that the method's load-bearing regularizer is under-specified and its Mamba proxy is unvalidated. These are fixable with a precise loss definition and an additional validation experiment, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a credible empirical paper that deserves a serious referee. The central claim—that a meta-learned Mamba can match or beat Transformers on MCL benchmarks while keeping a fixed-size state—is supported by the tables. The first contribution is real: nobody has formulated and meta-learned Mamba for MCL, and the selectivity regularizer that bridges SSM parameters to attention patterns is a genuine new technique, even if it is a variation on attention-supervision.\n\nWhat the paper does well: the evaluation is unusually thorough. Multiple datasets, fine-grained and domain-shift settings, length generalization, noise robustness, ablations on λ, state size, learning rate, and architecture variants, plus a comparison where the same regularization is applied to all models. The length generalization results (Fig. 4) are the most interesting finding: Mamba degrades much less than Transformers on longer streams. That is a concrete, falsifiable observation that goes beyond 'Mamba works.'\n\nSoft spots, in order of seriousness. First, the selectivity regularizer is under-specified and its mechanistic justification is shaky. The paper computes q = C_t B_j^T and treats it as Mamba's association pattern, but the actual state update in Eq. (3) includes the recurrent A_t factors. The surrogate is exact only if A = I, which contradicts the gating role of A. The stress-test note is right that the regularizer is load-bearing, because Appendix C.1 shows Mamba doesn't converge without it. But I don't think this sinks the paper. The accuracy numbers stand regardless of the exact mechanism; the regularizer demonstrably helps training, and the claim that it 'guides' Mamba can be true even if the proxy is imperfect. Still, the authors should be asked to specify how the KL is computed (the scores are unnormalized real numbers), and to validate the proxy, e.g., by comparing against full unrolled gradients or a diagnostic where A is ablated or included.\n\nSecond, no code is released, and the paper lacks a non-parametric baseline (e.g., nearest-neighbor in the frozen feature space). Such a baseline would tell us how much of the MCL performance comes from the learned dynamics versus the quality of the frozen CLIP features. This is important for calibrating the efficiency claims.\n\nThird, a smaller issue: the associative visualization for Mamba uses the same unvalidated proxy, so it is not independent evidence of mechanism. The paper is careful not to over-claim, but readers should treat Fig. 3 and the Appendix D figures as illustrative, not mechanistic.\n\nThe citation pattern looks fine; the related work is appropriate, and the self-citations are relevant. The limitation statement is thin, but not misleading. I'd send this to review. The authors should release code, clarify and validate the regularizer, and add the non-parametric baseline before acceptance; the core empirical finding is worth keeping.","headline":"A credible empirical case that Mamba works for MCL, held back by an under-specified regularizer and missing code.","tokens_in":22189,"tokens_out":2552,"would_cite":true,"duration_ms":23383,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba, a selective state space model with a fixed-size hidden state, can be meta-learned as a continual learner that matches or beats Transformers at lower memory and compute.","keywords":["meta-continual learning","state space models","Mamba","sequence prediction","linear attention","continual learning","selectivity regularization"],"falsifier":"Compare the learned association maps of a model trained with the paper's regularizer against a version trained with a full-path regularizer that accumulates each past token's influence through the actual recurrence by summing over all paths the product of intervening A matrices times C_t B_j^T (computed by auto-differentiation or by ablating individual B_j). If the two maps disagree sharply, or if the proxy-regularized model cannot predict which individual past token most changes the output (test by zeroing each B_j in turn and measuring the output change), the proxy assumption fails.","tokens_in":1848,"feed_emoji":"🧠","tokens_out":4948,"duration_ms":86863,"temperature":0.7,"pith_summary":"Continual learning can be framed as sequence prediction: a meta-learned model reads a stream of (x, y) pairs and predicts the label for a query. Transformer-based meta-continual learners do this well but must keep a growing cache of all past representations, which conflicts with continual learning's goal of not storing everything. This paper asks whether Mamba, an attention-free state space model with a fixed-size hidden state, can play the same role. It proposes MambaCL, which meta-learns Mamba across continual-learning episodes, plus a selectivity regularizer that steers training by linking Mamba's internal parameters to the attention patterns of Transformers. The paper reports that MambaCL matches or exceeds Transformer accuracy on most benchmark and generalization settings while using roughly half the parameters and running more than twice as fast.","feed_headline":"Fixed-memory Mamba matches Transformers at continual learning","feed_subtitle":"Meta-trained SSM learns from streams with a constant-state cache, beating cache-heavy Transformers on long sequences and noise.","key_machinery":"The central object is the associative indicator q^Mamba_{2t+1} = [C_{2t+1} B_j^T]_{j=1}^{2t}, built from Mamba's input-dependent output and input projection parameters C and B. It is used exactly like Transformer attention weights: a ground-truth indicator vector marks which past samples share the query's class, and a KL divergence steers Mamba's selectivity pattern toward it. This works because, under the duality identified between Mamba and linear attention, C_t plays the role of the query and B_j the role of a key, so C_t B_j^T encodes how much past token j should be retrieved when predicting at step t. The A_t recurrence that also shapes the hidden state is set aside in this proxy, and the fixed-size hidden state H_t itself is what gives Mamba its constant memory.","core_discovery":"The central claim is that selective state space models can serve as effective meta-continual learners once they are properly formulated and regularized. Concretely, the paper treats each class-incremental episode as a token sequence (x1, y1, ..., xt, yt, xtest) -> ytest, meta-learns a Mamba model over many such episodes, and introduces a selectivity regularization loss that forces the model's input-dependent SSM parameters to associate each query with the samples of the same class. The regularizer works by identifying the cross-products C_t B_j^T of Mamba's selective parameters with the query-key products of linear attention, which in turn approximate softmax attention; the KL divergence between this association pattern and the ground-truth same-class indicator stabilizes meta-training, which otherwise fails to converge for Mamba. Across general, fine-grained, and domain-shifted image classification plus regression tasks, the meta-learned Mamba matches or beats a vanilla Transformer on meta-test accuracy while using a fixed-size hidden state, and degrades far less than Transformers when meta-test episodes are longer, noisier, or from a held-out domain.","pith_inferences":["If the C_t B_j^T proxy is faithful, the same KL-regularizer recipe should extend to any recurrent or linear-attention model whose state update can be written as an outer-product accumulation; testing it on other recurrent architectures with fixed-size states would show whether Mamba's advantage is specific to selective SSMs or a general property of recurrent compression.","The strong length generalization suggests that a meta-learned recurrent learner may be inherently less prone to positional overfitting than attention models, because it must represent content in a capacity-limited state; this predicts that MambaCL should also transfer to streaming settings with unpredictable task boundaries, which the paper does not test.","The random-token vocabulary trick, where class identities are arbitrary symbols within each episode, means the learner is effectively trained to handle any label permutation; a natural extension the paper does not explore is using MambaCL as a task-agnostic few-shot classifier on streams where the label set changes by example rather than by task.","The paper's stated limitations leave offline CL and larger-scale datasets unexplored; one concrete implication of the results is that Mamba's fixed-state compression could bring replay-free offline CL within reach if the selectivity regularizer transfers to multi-epoch streams."],"forward_implications":["MambaCL matches or exceeds Transformer meta-test accuracy on general, fine-grained, and domain-shift benchmarks while keeping a fixed-size state, removing the memory growth that conflicts with continual learning's goal of not storing all seen samples.","It generalizes to meta-test episodes with ten times as many shots as training (50 versus 5) with only about a 10% accuracy drop, while Transformer and Linear Transformer degrade sharply at untrained episode lengths.","Under Gaussian input noise up to σ = 10, Mamba's accuracy remains far more stable than Transformer's, and on the DomainNet held-out-domain test it reaches 55.6% on quickdraw versus 50.2% for Transformer.","The selectivity regularizer is necessary for Mamba to converge at all: without it the meta-training loss oscillates and stops decreasing, while varying the regularization weight λ across 0.1 to 2.0 leaves results stable.","The efficiency claim is concrete: MambaCL uses 5.4M parameters and runs 858 episodes per second versus the Transformer's 9.2M parameters and 325 episodes per second, at comparable or better accuracy."],"supporting_citations":[{"why":"Supplies Mamba, the selective state space model that the paper meta-learns as its continual learner.","marker":"[16]"},{"why":"Establishes the meta-continual-learning-as-sequence-modeling formulation and the Transformer baseline setup the paper adopts.","marker":"[32]"},{"why":"Provides the duality between SSMs and linear attention that grounds the selectivity regularizer's query-key interpretation of C and B.","marker":"[9]"},{"why":"Introduces linear attention with recurrent state, the bridge model used to derive the associative indicators for Mamba.","marker":"[27]"},{"why":"Defines the vanilla Transformer whose key-value cache the paper compares against.","marker":"[65]"},{"why":"Supplies Performer, the other attention-free baseline used in comparisons.","marker":"[8]"},{"why":"Provides OML, the SGD-based meta-continual-learning baseline.","marker":"[25]"}],"fun_headline_variants":["Mamba learns continually with fixed memory, no cache growth","Attention-free Mamba rivals Transformers in continual learning","MambaCL: meta-learned SSM beats Transformers on longer streams","Meta-learned Mamba matches Transformer accuracy with constant-state cache","Selective SSM achieves continual learning with fixed-size memory"],"cache_read_input_tokens":24448,"weakest_assumption_plain":"The load-bearing assumption is that the cross-products C_t B_j^T of Mamba's input-dependent SSM parameters faithfully represent how much Mamba actually retrieves each past token, even though the recurrent A_t matrices that multiply earlier B_j values are dropped from the regularization; if that proxy is unfaithful, the selectivity regularizer may teach the wrong association pattern and the reported gains may not transfer to other MCL definitions or architectures.","fun_headline_variants_meta":{"raw":{"variants":["Mamba learns continually with fixed memory, no cache growth","Attention-free Mamba rivals Transformers in continual learning","MambaCL: meta-learned SSM beats Transformers on longer streams","Meta-learned Mamba matches Transformer accuracy with constant-state cache","Selective SSM achieves continual learning with fixed-size memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1692,"prompt_tokens":1077,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":693,"tokens_out":615,"duration_ms":5702,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:00:40.788481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the learned association maps of a model trained with the paper's regularizer against a version trained with a full-path regularizer that accumulates each past token's influence through the actual recurrence by summing over all paths the product of intervening A matrices times C_t B_j^T (computed by auto-differentiation or by ablating individual B_j). If the two maps disagree sharply, or if the proxy-regularized model cannot predict which individual past token most changes the output (test by zeroing each B_j in turn and measuring the output change), the proxy assumption fails.","supporting_citations":[{"cited_title":"Meta-learning representations for continual learning","cited_arxiv_id":null,"evidence_quote":"Provides OML, the SGD-based meta-continual-learning baseline."},{"cited_title":"Recasting continual learning as sequence mod- eling","cited_arxiv_id":null,"evidence_quote":"Establishes the meta-continual-learning-as-sequence-modeling formulation and the Transformer baseline setup the paper adopts."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Defines the vanilla Transformer whose key-value cache the paper compares against."}],"review_version":1}