{"id":"8594d0c9-2003-4d47-a851-266f62cff6a2","arxiv_id":"2411.15469","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Mamba-CL derives four orthogonality conditions for Mamba's SSM parameters and enforces them via null-space projection, yielding state-of-the-art continual learning accuracy on ImageNet-R, CIFAR-100, and DomainNet.","lead":"Mamba-CL is a method for continually training a Mamba vision model on a sequence of classification tasks without catastrophic forgetting. It works by constraining parameter updates to be orthogonal to the feature subspace of previous tasks, and it outperforms prior continual-learning methods on four benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq.(3)-(4) treats the selective Mamba forward pass as a fixed-kernel convolution, but B, C, δ vary per token in Eq.(1)-(2), so the derivation of Eq.(25) from K_t = K_{t+1} does not establish the claimed output-consistency guarantee.","rationale":"The paper's headline contribution is a theoretical guarantee: Eq.(25) is claimed sufficient for Eq.(7). Following the proof from Eq.(8) to Eq.(25), the first non-trivial step is Eq.(4), and it is wrong for selective SSMs: B, C, and δ are position-dependent, so no fixed convolutional kernel exists. This is not a matter of consensus or approximation; it contradicts the model described in Eq.(1)-(2). If Eq.(4) fails, K_t = K_{t+1} is undefined and the factorization in Eq.(10)-(11) cannot be used to derive Eq.(25). I therefore cannot certify the central claim as stated. The reader's weakest assumption about Eq.(12)-(13) is real, but it occurs downstream; even a correct treatment of powers would not rescue the proof because the kernel representation itself is invalid. The empirical results are extensive and the ablations show each projector contributes, so the work should not be rejected outright. It needs a corrected theoretical derivation: either prove consistency via the true time-varying recurrence, or explicitly restrict the theoretical claim to a non-selective/LTI simplification and validate empirically that Mamba's selectivity is not the source of the gains. This is a major revision rather than a minor caveat, and I would keep the CONDITIONAL verdict while strengthening the required revisions.","tokens_in":21927,"tokens_out":15437,"duration_ms":145630,"concrete_test":"Run the released De-focus Mamba block on a random input sequence X of length L>1, using the actual selective scan to compute y_selective. Also build a fixed kernel K from the first-token parameters, e.g., K = [C_1 B_1, C_1 A_bar_1 B_1, ..., C_1 A_bar_1^k B_1], and compute y_conv = X * K. If y_selective ≠ y_conv whenever δ_l, B_l, C_l vary over l, Eq.(4) is falsified and the consistency proof in Section 4.2 needs a different starting point. This single computational check directly tests whether the paper's kernel representation exists for the model it analyzes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The proof starts from Eq.(8)-(9): old outputs are preserved if the convolutional kernel is unchanged, K_t = K_{t+1}. This presumes Eq.(4), Y = X * K, with K defined in Eq.(3). For the Mamba SSM described in Eq.(1)-(2), B, C, and δ are functions of the input and vary across the sequence: B ∈ R^{L×N}, C ∈ R^{L×N}, δ ∈ R^{L×1}, and A_bar_l = exp(δ_l A). The recurrence is h_l = A_bar_l h_{l-1} + B_bar_l x_l, y_l = C_l h_l, whose expansion is y_l = Σ_{j≤l} C_l (Π_{m=j+1}^{l} A_bar_m) B_bar_j x_j. This is a time-varying lower-triangular map, not a convolution with a single kernel whose taps are C A^k B. Thus Eq.(4) is false for the selective SSM this paper uses. The entire reduction to factor-wise conditions A^k_t = A^k_{t+1}, B_t = B_{t+1}, C_t = C_{t+1}, and then to Eq.(25), depends on this non-existent kernel. Even if all four projections in Eq.(27) were applied exactly, the argument from Eq.(8)-(9) does not go through. The reader's Eq.(12)-(13) objection is a separate, later manifestation of the same over-simplification: A^k in Eq.(10) is a matrix/tensor power of a per-token discretized operator, not an elementwise scalar power of exp(x), so monotonicity cannot justify δA equality. The approximate-null-space implementation does not repair either flaw; it makes the guarantee approximate in a second, independent way.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mamba-CL, a rehearsal-free class-incremental learning method that fine-tunes the SSM parameters of a pre-trained Mamba vision backbone under null-space orthogonality constraints. The claimed contribution is a theoretical guarantee that the SSM output on previous-task inputs is unchanged after learning a new task, obtained by deriving four sufficient conditions in Eq. (25) on updates to A, W_B, W_C, and W_delta, and implementing those conditions through SVD-based projectors. The experimental part compares Mamba-CL with prompt- and adapter-based CL methods on ImageNet-R, CIFAR-100, and DomainNet splits, including 50- and 100-task settings, and reports consistent accuracy improvements over the Mamba-Seq baseline and prior ViT-based methods.","tokens_in":22273,"tokens_out":5629,"duration_ms":53526,"significance":"If the consistency guarantee were valid, the paper would be a valuable extension of null-space orthogonal-projection methods to SSM architectures, and the extensive experiments (Tables 1-5, ablation Table 3, runtime analysis in Section 5.5) provide credible evidence of empirical benefit. The paper also reports code availability and ablates each projector individually. However, the theoretical derivation that underpins the central claim is not sound, and the claimed guarantee does not follow from the stated equations. Consequently, the significance of the method as a theoretically grounded continual learning approach is not established, although the empirical observations may remain useful.","major_comments":[{"comment":"For the selective SSM described in Eqs. (1)-(2), the forward map is not a fixed-kernel convolution. Since B_l, C_l, and delta_l are functions of the input token x_l, the recurrence h_l = A_bar_l h_{l-1} + B_bar_l x_l, y_l = C_l h_l expands to y_l = sum_{j<=l} C_l (product_{m=j+1}^l A_bar_m) B_bar_j x_j, which is a time-varying lower-triangular operator rather than X*K with a single kernel K = [CB, CAB, CA^2 B, ...]. Therefore Eq. (9), K_t = K_{t+1}, is neither necessary nor sufficient for Eq. (7), and the derivation of Eq. (25) does not establish output consistency for the model actually used. This is the load-bearing step of the paper's theoretical claim.","section":"Section 3.1, Eq. (4)"},{"comment":"The inference from [exp(delta_t A_t)]^k = [exp(delta_{t+1} A_{t+1})]^k to delta_t A_t = delta_{t+1} A_{t+1} is unjustified. In Eq. (2), A is defined as exp(delta A) element-wise; delta is a vector and A is a D by N tensor, so the exponent is not a scalar, and the monotonicity argument used for [exp(x)]^k does not apply to this tensor product structure. Moreover, even for a scalar argument, equality of all powers only gives equality of the positive exponentials, not directly the equality asserted in Eq. (13). This gap is load-bearing because the subsequent decomposition into delta_t = delta_{t+1} and delta_t Delta A = 0 in Eq. (14) relies on Eq. (13).","section":"Section 4.2.1, Eqs. (12)-(13)"},{"comment":"Even if Eq. (25) were sufficient for output consistency, the implementation does not enforce Eq. (25) exactly. The projection matrices are built from an approximate null space selected by a corner-point threshold, and the balance factor eta in Section 4.3 replaces H by eta H + (1-eta) I with eta < 1 in the reported experiments. Consequently, the parameter updates actually applied in the experiments do not satisfy the four conditions of Eq. (25), so the abstract's claim that the method 'theoretically guarantees' output consistency is not supported by the implementation. The experiments can still be read as evaluating an approximate version of the idea, but not the exact theoretical guarantee stated in the paper.","section":"Section 4.3, Eq. (27) and the eta relaxation"}],"minor_comments":[{"comment":"The symbol A is used both for the continuous state-transition parameter and for the discretized parameter exp(delta A) in Eq. (2), and the same overloaded notation reappears in Eqs. (10)-(14). Using distinct symbols (for example, A and A_bar) would remove substantial ambiguity from the consistency derivation.","section":"Section 3.1, Eq. (2)"},{"comment":"There is a typo: 'Cosidering' should be 'Considering'.","section":"Section 4.2.2"},{"comment":"The name 'VMambda' in Section 2.1 and the inconsistent spelling 'CIF AR-100' throughout should be corrected to 'VMamba' and 'CIFAR-100'.","section":"Related Work and Experiments"},{"comment":"The text says that the datasets are 'also split into 50 or 100 tasks' after having described only four benchmarks; the long-sequence results in Table 2 are clear, but the preceding sentence should specify which benchmarks receive the 50- and 100-task splits.","section":"Section 5.1"}],"recommendation":"reject","confidential_remarks":"The central theoretical contribution is invalid for the selective SSM used in the experiments: the forward pass is not a fixed-kernel convolution, so the kernel-consistency argument leading to Eq. (25) does not establish Eq. (7). This is not a local fix; it concerns the core mechanism claimed for anti-forgetting. The empirical results are extensive and may justify a future empirical paper with a weaker claim, but as written the manuscript's headline theoretical guarantee is unsupported. The direct predecessor VPT-NSP2 is cited, and the novelty rests on transferring that null-space idea to Mamba, which is exactly where the derivation fails."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of Mamba-CL.\n\nThe paper does something useful empirically: it takes the null-space projection idea from CNN/ViT continual learning and adapts it to Mamba backbones. The experiments are substantial and well run — four benchmarks, long-sequence settings, several Mamba variants, and ablations that show each projection matters. The gains over the fine-tuning baseline are large and consistent. If this were just an empirical method for Mamba continual learning, it would be a decent contribution.\n\nThe problem is the theoretical derivation that supposedly guarantees the method. Section 4.2 builds on Eq.(4), which treats the Mamba forward pass as a global convolution with a fixed kernel K = [CB, CAB, …]. That is true for non-selective SSMs with input-invariant B and C. But the paper's own Eq.(1)–(2) define B, C, and δ as input-dependent, varying across token positions. The actual recurrence is time-varying: the output at position l is a sum over j of C_l (∏ A_bar_m) B_j x_j, not a convolution with a single kernel. So Eq.(4) is false for the architecture this paper uses, and everything else — the reduction K_t = K_{t+1}, the factor-wise conditions, and the four projections in Eq.(25) — does not establish the claimed output-consistency guarantee. Even if all four projections were exact, the argument from Eq.(8)–(9) does not go through.\n\nThe reader's objection about the element-wise exponential and monotonicity in Eq.(12)–(13) is real but secondary; the root problem is earlier. The approximate null-space implementation only makes the guarantee approximate in a second, independent way.\n\nSo the abstract's claim of a 'theoretical guarantee' is not supported. The method may still work as a well-motivated heuristic, and the empirical evidence says it does. But the paper needs a major revision: either derive conditions for the actual time-varying selective SSM, or drop the formal guarantee claim and present the orthogonality as an approximate stability mechanism.\n\nFor peer review: this deserves referee time because the empirical contribution is significant and the flaw is subtle and instructive. But it should not be accepted as is.","headline":"Strong empirical results on Mamba continual learning, but the theoretical guarantee rests on a false convolution assumption for selective SSMs.","tokens_in":22865,"tokens_out":4223,"would_cite":false,"duration_ms":37315,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba-CL claims that four null-space orthogonality constraints on the SSM parameters let a Mamba model learn new tasks without changing its outputs on old tasks, and reports state-of-the-art accuracy on four class-incremental benchmarks.","keywords":["continual learning","catastrophic forgetting","state space models","Mamba","null-space projection","orthogonal gradient projection","class-incremental learning","selective state spaces"],"falsifier":"Choose random $\\delta_1, A_1, \\delta_2, A_2$ and test whether $[\\exp(\\delta_1 A_1)]^k = [\\exp(\\delta_2 A_2)]^k$ for $k=1,2,3$ implies $\\delta_1 A_1 = \\delta_2 A_2$; any counterexample invalidates the derivation step in Eqs. (12)-(13). A direct check is to take a trained Mamba-CL model, verify that the four conditions in Eq. (25) hold exactly for a stored $X_t$, and measure whether $\\mathrm{SSM}(X_t \\mid \\theta_t) - \\mathrm{SSM}(X_t \\mid \\theta_{t+1})$ is zero to numerical precision.","tokens_in":21681,"feed_emoji":"🧠","tokens_out":8452,"duration_ms":68799,"temperature":0.7,"pith_summary":"The paper proposes Mamba-CL, a continual-learning method for the Mamba selective state-space model. The central claim is that catastrophic forgetting can be prevented by updating only the SSM core under four sufficient orthogonality conditions: $\\delta_t \\Delta A = 0$, $X_t \\Delta W_\\delta = 0$, $\\delta_t X_t \\Delta W_B = 0$, and $X_t \\Delta W_C = 0$. If these hold, the SSM output on any previous-task input $X_t$ is the same before and after learning a new task, so old knowledge stays intact. The method enforces the conditions with null-space projectors built from previous-task feature covariances, and reports accuracy improvements over prior continual-learning baselines on four class-incremental benchmarks. This matters because it extends the null-space anti-forgetting recipe from convolutional and transformer layers to the recurrent, non-linear discretization inside Mamba.","feed_headline":"Four null-space rules stop Mamba from forgetting old tasks","feed_subtitle":"Projecting SSM updates orthogonal to previous-task features preserves outputs and lifts accuracy on four continual benchmarks.","key_machinery":"The load-bearing machinery is the SSM's convolutional-kernel representation. Since $Y = X * K$, the consistency objective Eq. (7) is turned into a kernel equality $K_t = K_{t+1}$. The kernel factors are the discretized transition matrix $\\bar{A} = \\exp(\\delta A)$, the input projection $\\bar{B}$ derived through zero-order hold, and the output projection $C$; the paper argues that enforcing $\\bar{A}_t^k = \\bar{A}_{t+1}^k$, $\\bar{B}_t = \\bar{B}_{t+1}$, and $C_t = C_{t+1}$ element-wise is sufficient for kernel consistency. The reduction to Eq. (25) uses monotonicity of the element-wise exponential and of the softplus discretization step to convert these equalities into orthogonality conditions on $\\Delta A$, $\\Delta W_B$, $\\Delta W_C$, and $\\Delta W_\\delta$. These conditions are implemented by SVD-based projectors $H_1, H_2, H_3$ onto approximate null spaces of the previous-task feature covariance matrices.","core_discovery":"Mamba-CL's central claim is that output consistency for an SSM module reduces to four independent parameter constraints. Because the forward pass is a global convolution $Y = X * K$ with kernel $K = [CB, CAB, CA^2B, \\ldots]$, keeping old outputs unchanged is equivalent to keeping the kernel unchanged. The paper decomposes kernel consistency into consistency of $\\bar{A}^k$, $\\bar{B}$, and $\\bar{C}$, and, using the zero-order hold discretization $\\bar{A} = \\exp(\\delta A)$ and $\\bar{B} = (\\delta A)^{-1}(\\exp(\\delta A - I))\\delta B$, derives the four sufficient conditions in Eq. (25). It then constructs three null-space projectors from uncentered covariance matrices of $X_t$, $\\delta_t$, and $\\delta_t X_t$, so that the projected gradients satisfy the conditions. On 10- and 20-split ImageNet-R, 10-split CIFAR-100, and 10-split DomainNet, plus 50- and 100-task versions, Mamba-CL reports higher final accuracy and lower forgetting than sequential fine-tuning and than prior prompt- and adapter-based methods.","pith_inferences":["A natural follow-up the authors left implicit is to weaken the sufficiency assumptions: because the conditions are sufficient, not necessary, there may be larger allowable update directions that preserve old outputs and improve plasticity.","The paper's derivation assumes an element-wise-exponential reduction of $[\\exp(\\delta A)]^k$; a corrected treatment of $\\delta$ as a vector and $A$ as a tensor would also transfer to other selective SSMs that share Mamba's discretization.","The guarantee is approximate in practice because the projectors use an approximate null space selected by a corner-point criterion; measuring output drift as a function of the number of discarded singular values would quantify how much of the reported stability comes from the approximation.","One could extend the method to streaming-task boundaries by updating the uncentered covariance matrices incrementally instead of recomputing them once per task; the derivation itself does not require clean task boundaries."],"forward_implications":["If the four conditions hold, a previous-task input produces identical SSM outputs before and after each new task, so old representations are not drifted and no replay buffer is needed.","The reported gains on the four class-incremental benchmarks and on 50/100-task long sequences suggest the recipe keeps working when the number of tasks grows.","The ablation shows the $\\delta$ projection contributes most, because $\\delta$ controls discretization of both $A$ and $B$, while the $A$ projection contributes least.","The extra training cost is modest (about 7% more runtime on average) and the additional memory is constant in task count, so the method does not inherit the storage growth of rehearsal or expansion methods."],"supporting_citations":[{"why":"Defines the Mamba selective SSM block, its four parameters A, B, C, and delta, and the ZOH discretization the paper's derivation starts from.","marker":"Gu and Dao (2023)"},{"why":"Supplies the De-focus Mamba backbone and its ImageNet-21k pretrained weights used as the base model in the main experiments.","marker":"Tao et al (2024)"},{"why":"Provides the null-space training method and the adaptive zero-singular-value selection used to build projectors.","marker":"Wang et al (2021)"},{"why":"Extends null-space projection to prompt tuning in ViTs with sufficient consistency conditions, the direct precedent Mamba-CL adapts.","marker":"Lu et al (2024)"},{"why":"Contributes gradient projection memory, the projection-based continual-learning framework the optimizer in Eq. (26) follows.","marker":"Saha et al (2021)"},{"why":"Establishes the orthogonal-weight-modification idea of projecting gradients into the null space of previous inputs.","marker":"Zeng et al (2019)"}],"fun_headline_variants":["Mamba-CL: null-space tricks keep old tasks intact","Null-space projection lets Mamba learn forever","Anti-forgetting Mamba: project updates away from old features","Mamba doesn't forget with null-space parameter updates","Continual Mamba: orthogonality stops catastrophic forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole guarantee rests on assuming that equal powers of the discretized transition matrices force the products $\\delta_t A_t$ and $\\delta_{t+1} A_{t+1}$ to be equal, even though $\\delta$ is a vector and $A$ a tensor and the exponential is element-wise; if that step fails, the four projection conditions do not actually guarantee unchanged outputs.","fun_headline_variants_meta":{"raw":{"variants":["Mamba-CL: null-space tricks keep old tasks intact","Null-space projection lets Mamba learn forever","Anti-forgetting Mamba: project updates away from old features","Mamba doesn't forget with null-space parameter updates","Continual Mamba: orthogonality stops catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1394,"prompt_tokens":1010,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":626,"tokens_out":384,"duration_ms":4392,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:16:04.732613+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Choose random $\\delta_1, A_1, \\delta_2, A_2$ and test whether $[\\exp(\\delta_1 A_1)]^k = [\\exp(\\delta_2 A_2)]^k$ for $k=1,2,3$ implies $\\delta_1 A_1 = \\delta_2 A_2$; any counterexample invalidates the derivation step in Eqs. (12)-(13). A direct check is to take a trained Mamba-CL model, verify that the four conditions in Eq. (25) hold exactly for a stored $X_t$, and measure whether $\\mathrm{SSM}(X_t \\mid \\theta_t) - \\mathrm{SSM}(X_t \\mid \\theta_{t+1})$ is zero to numerical precision.","supporting_citations":[],"review_version":1}