{"id":"da36ff46-eafc-4bee-ae41-cd2150d1d5f1","arxiv_id":"2411.15224","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Proposes ProDiaL, a PEFT method for Mamba that tunes Projector weights via a block-diagonal linear transformation plus a low-rank residual, based on the claim that Projectors dominate transfer learning.","lead":"This paper argues that in Mamba models, the linear Projector layers, not the state-space core, matter most for adapting to new tasks, and introduces ProDiaL, a parameter-efficient fine-tuning method that updates only a near-diagonal transform of the Projector weights.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own ablation (Tab. 7) shows the off-diagonal LoRA term ε, not the diagonal-centric Db, drives ProDiaL's gains; the central diagonal-centric claim is not load-bearing.","rationale":"I started from the reader's weakest assumption about T being near-diagonal and found a stronger internal problem: the paper's own ablation table undercuts the method's namesake. Even if T were consistently near-diagonal, the reported gains do not come from the diagonal-centric component. The off-diagonal LoRA term ε is responsible for the improvement, and standard LoRA on the same projectors is nearly as good as full ProDiaL. This matters because ProDiaL's novelty is precisely the diagonal-centric parameterization; without it, the contribution reduces to the projector-importance finding, which is plausible and independently supported by the component ablations in Tabs. 1 and 2. The existing CONDITIONAL verdict is appropriate, but the condition should explicitly include the matched-parameter ablation above, along with the reader's requests for code, error bars, and corrected parameter claims. If the ablation shows that removing Db does not hurt, the paper should be rejected rather than conditionally accepted.","tokens_in":21579,"tokens_out":11436,"duration_ms":116945,"concrete_test":"Run a three-arm, matched-parameter ablation on Caltech and HellaSwag: (a) LoRA on projectors only, (b) ProDiaL with Db removed (W' = sW + ε), and (c) full ProDiaL, over at least three seeds, reporting mean and std. If (b) matches (c) within noise, the diagonal-centric block Db is unnecessary and the central claim fails; if (b) is clearly worse than (c), the diagonal-centric term is required.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that fine-tuned Mamba projectors are obtained from pretrained W by a near-diagonal linear transformation, so it suffices to train the block-diagonal Db and a low-rank residual ε (Eq. 7). This claim is contradicted by the paper's own numbers. On Caltech (Tab. 7), Proj-Db alone gives 95.24 with 0.57M parameters; adding ε, a LoRA-style off-diagonal update, gives 96.16; adding s gives 96.24. Thus ε contributes +0.92 points to Db, while the entire diagonal-centric machinery (Db plus s) contributes only +0.08 once ε is present. Meanwhile, ordinary LoRA on the same projectors (Tab. 3) reaches 96.01 with 0.63M parameters, which is above Db-only and within 0.23 points of full ProDiaL. The method therefore behaves essentially as LoRA on projectors; the diagonal-centric component is not the source of the reported performance. The supporting analysis in Sec. 4.2.1 is also weak: Tdet = W^-1 W' is near-identity for any fine-tuned W' close to W, since Tdet = I + W^-1(W'-W), and Fig. 2 shows a single layer on one dataset with no quantitative diagonal-energy measure. If the diagonal component is unnecessary, the methodological novelty of ProDiaL reduces to 'apply LoRA to projectors,' which the paper already lists as a baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies parameter-efficient fine-tuning (PEFT) for Mamba architectures. It makes two claims: (1) the linear Projectors in Mamba blocks, not the SSM modules, are the main contributors to transfer learning; (2) based on an observed near-diagonal linear relation W' ≈ W T between pretrained and fine-tuned projector weights, the paper proposes ProDiaL, which freezes W and trains a block-diagonal matrix Db, a low-rank off-diagonal term ε, and a scaling vector s, giving W' = s W Db + ε. The method is evaluated on Mamba-130M/370M/1.4B for language reasoning tasks and Vim-tiny/small for vision classification tasks, with comparisons to full fine-tuning, BitFit, Strong, LoRA, and DoRA. The paper reports that ProDiaL outperforms baselines while using a small fraction of the model parameters.","tokens_in":21876,"tokens_out":4811,"duration_ms":44377,"significance":"The finding that Projectors, rather than SSMs, are the primary drivers of downstream adaptation in Mamba is a useful empirical insight and is broadly supported by the component-level comparisons in Tables 1 and 2. If the ProDiaL parameterization were shown to be the source of the reported gains, the method would be a valuable addition to PEFT for Mamba. The paper also provides a breadth of experiments across vision and language models and model scales, and it includes ablation studies. However, as written, two load-bearing claims are not supported by the paper's own numbers: the 'less than 1%' parameter claim is factually contradicted by Tables 3 and 4, and the ablation in Table 7 indicates that the off-diagonal LoRA term, not the diagonal-centric block, drives most of ProDiaL's improvement over the projector-only baseline. These issues require substantial revision before the central methodological claims can be accepted.","major_comments":[{"comment":"The claim that ProDiaL fine-tunes 'less than 1% of the total parameters' is contradicted by the parameter counts in Tables 3 and 4. For Mamba-130M, the Both-Proj ProDiaL configuration uses 2.42M parameters, which is 1.86% of 130M; for Vim-tiny, 0.67M parameters is about 9.6% of the 7M-parameter model; for Mamba-370M and Mamba-1.4B, the reported values are about 1.55% and 3.6%, respectively. The efficiency claim should be corrected to state the actual fractions, or the experimental configurations should be changed so that the stated bound holds.","section":"Abstract and Section 7"},{"comment":"The ablation study does not support the paper's central claim that the diagonal-centric transformation is the key component. On Caltech, Proj-Db alone reaches 95.24 with 0.57M parameters, adding ε raises accuracy to 96.16, and adding s gives 96.24. Thus ε contributes +0.92 points once Db is present, whereas the entire diagonal-centric machinery (Db plus s) contributes only +0.08 points once ε is included. Moreover, ordinary LoRA on the same projectors reaches 96.01 with 0.63M parameters, which is already above the Db-only result and within 0.23 points of full ProDiaL. These numbers suggest that ProDiaL behaves essentially as 'LoRA on projectors' and that the diagonal-centric component is not the source of the reported gains. The authors need to either provide evidence that the diagonal structure is genuinely load-bearing or reframe the method's contribution accordingly.","section":"Table 7 and Section 5.5"},{"comment":"The evidence that the projector transformation T is near-diagonal is weak and partly circular. Since Tdet = W⁺W' = I + W⁺(W' - W), any fine-tuned W' that is close to W will produce a Tdet close to the identity, regardless of whether diagonal or off-diagonal structure is important for the task. The visualization in Fig. 2 shows only a single input projector at layer 4 on the Caltech dataset, and the supplementary text confirms this choice, so the claim that the pattern is 'consistently observed across both Projectors, all layers, and both Mamba LLM and Vision models' is not supported by the reported evidence. The paper should report a quantitative diagonal-energy measure across layers, projectors, and datasets, and ideally compare against a random or permutation baseline.","section":"Section 4.2.1, Eq. (5), and Fig. 2"},{"comment":"The hyperparameters rb1, rb2, and rε are selected per dataset, and Table S1 shows that the best configuration varies with the dataset and appears to be chosen with reference to test accuracy. Combined with the fact that the main tables report single runs (only Table S4 gives three-seed results), the reported differences between ProDiaL and the LoRA/DoRA baselines are not shown to be statistically reliable. For example, the Caltech gap between ProDiaL (96.24) and LoRA (96.01) is 0.23 points, which is smaller than the seed variation reported for similar settings in Table S4. The authors should fix hyperparameters on a validation split or report error bars across multiple seeds for all headline comparisons.","section":"Tables S6–S9 and Table S1"}],"minor_comments":[{"comment":"The sentence 'While state-space models (SSMs) have been regarded as the cornerstone of Mamba architecture, then expected to play a primary role in transfer learning' contains a grammatical error ('then expected' should likely be 'and are expected').","section":"Abstract"},{"comment":"The notation for the scaling vector s is ambiguous: s is defined as a vector in R^{d_out}, but it is used in the product s W Db without specifying whether it is a diagonal matrix, a row-wise scaling, or an outer product. Please define the broadcasting or use an explicit diagonal operator.","section":"Eq. (7) and Algorithm 1"},{"comment":"In Table 4, the parameter counts for the larger models are rounded to values such as 0.06B and 0.05B, which makes it difficult to verify the claimed parameter-efficiency ratios. Reporting exact parameter counts would allow readers to check the efficiency claims.","section":"Section 5.3 and Table 4"},{"comment":"The Flower results in Table S1 are inconsistent with the main text: Table S1 reports 86.96 for the (192,384,16) configuration and 89.12 for the (8,32,8) configuration, while Table 3 reports 88.00 for ProDiaL on Flowers and Table S8 lists (8,32,8). Please clarify which hyperparameter configuration corresponds to the results in Table 3.","section":"Appendix B, Table S1"},{"comment":"The algorithm constructs Da = diag(x1, ..., xrb), while Eq. (7) uses Da = diag(x1, ..., xn). The notation n and rb should be unified.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The projector-dominance finding is empirically useful and well supported by the component ablations, but the paper's headline method claim rests on the diagonal-centric transformation, which the authors' own ablation undermines. The 'less than 1%' statement is also factually incorrect for the reported configurations. I would encourage the editor to treat the requested revisions as substantive: the authors should either provide quantitative evidence that diagonal structure matters or reposition the contribution as a projector-targeted PEFT study with LoRA as the effective component."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe headline: the useful empirical finding here is that Mamba's projectors, not the SSM, carry most of the transfer-learning load in PEFT. The proposed ProDiaL method is a reasonable LoRA variant, but the paper's own ablation undercuts the \"diagonal-centric\" story, and several claims are overstated.\n\nWhat's new and good: Tables 1 and 2 give a clean component ablation across a vision Mamba and a 130M Mamba LLM, and the projector-dominance result is consistent in both. That is a genuinely useful observation for the PEFT community and matches the intuition that the gating and linear projection layers are where task-specific adaptation happens. ProDiaL is simple to implement, and the trick of merging the trained transformation back into the frozen W is practical. There is also a three-seed table in the supplementary (Table S4), which is more than many PEFT papers bother to run.\n\nWhere it gets soft: the \"less than 1%\" parameter claim is flatly contradicted by their own tables—0.65M on a 7M Vim-tiny is about 9%, and 2.42M on a 130M Mamba is about 1.9%. That needs correcting. Main results have no error bars; the reported ProDiaL gains over LoRA/DoRA are 0.2 to 0.8 points, which is within the noise of the three-seed runs in Table S4. The Mamba LLM evaluation selects the checkpoint with the highest test accuracy, which is a form of test-set tuning, and the ProDiaL hyperparameters (rb1, rb2, r_epsilon) are picked per dataset, apparently on test accuracy as well. That makes the comparison generous to ProDiaL.\n\nMost importantly, the central \"diagonal-centric\" claim is not supported by their own ablation (Table 7). On Caltech, Proj-Db alone gives 95.24; adding the off-diagonal LoRA term epsilon gives 96.16; adding the scaling s gives 96.24. So epsilon contributes +0.92, while the entire diagonal-centric machinery (Db and s) contributes only +0.08 once epsilon is present. Meanwhile, ordinary LoRA on the same projectors reaches 96.01 with similar parameter counts. The method behaves essentially as LoRA on projectors; the diagonal block is not the source of the reported performance. The analysis in Section 4.2.1 is also weak: T_det = W^-1 W' is near-identity whenever W' is close to W, and Fig. 2 shows one layer on one dataset without any quantitative diagonal-energy measure. The \"first study\" sentence in Section 5.2 is also wrong given Halloran et al. [17] is cited and compared.\n\nWho this is for: researchers working on PEFT for SSM-based models. The projector-dominance result deserves to be known and checked. The method as presented needs more work before I would trust the specific numbers.\n\nRecommendation: worth sending to peer review, but with major revisions: correct the parameter claim, report error bars, drop the test-set checkpoint selection, and reframe the method honestly as \"LoRA on projectors with a block-diagonal rescaling.\" The diagonal-centric framing should be demoted to a secondary observation.","headline":"The projector-dominance finding is the real contribution; ProDiaL is essentially LoRA on projectors with a diagonal wrapper, and the paper's own ablation contradicts the diagonal-centric narrative.","tokens_in":22473,"tokens_out":3919,"would_cite":true,"duration_ms":34112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba's transfer learning is carried by its linear projectors, not its state-space core, and a diagonal-centric transform of those frozen projectors tunes the model with under 1% of parameters.","keywords":["Mamba","parameter-efficient fine-tuning","state-space models","Vision Mamba","projector fine-tuning","diagonal-centric transformation","low-rank adaptation","transfer learning"],"falsifier":"Compute $T_{\\mathrm{det}} = W^{-1}W'$ for all projector layers on a held-out task and compare the L1 mass of off-diagonal entries with the diagonal; then train ProDiaL with the off-diagonal term removed. If removing $\\epsilon$ changes accuracy by more than the method's reported gains, or if a synthetic task with a known rotation $W' = W R$, where $R$ is a 45-degree rotation matrix, causes accuracy to collapse, the diagonal-centric assumption is falsified.","tokens_in":21340,"feed_emoji":"🎯","tokens_out":10862,"duration_ms":86836,"temperature":0.7,"pith_summary":"The paper argues that in Mamba architectures, transfer learning is carried by the two linear projectors—the input and output linear layers that wrap the selective state-space model (SSM)—rather than by the SSM itself. Component-wise fine-tuning experiments on Mamba LLMs and Vision Mamba show that tuning a single projector outperforms tuning the SSM's own parameters. Based on the observation that a fine-tuned projector weight $W'$ can be written as $W T$ with $T$ close to the identity and dominated by its diagonal, the authors propose ProDiaL: freeze $W$ and train $W' = s W D_b + \\epsilon$, where $D_b$ is block-diagonal, $s$ is a per-output scaling vector, and $\\epsilon$ is low-rank. ProDiaL uses less than 1% of total parameters and reports accuracy higher than or comparable to full fine-tuning, LoRA, and DoRA across language reasoning and image classification benchmarks, at several model sizes.","feed_headline":"Projectors, not SSMs, carry Mamba transfer learning","feed_subtitle":"A diagonal-centric transform of frozen projector weights tunes Mamba vision and language models with under 1% of parameters.","key_machinery":"The central object is the near-diagonal linear transformation $T$ between pretrained and fine-tuned projector weights, recovered as $T_{\\mathrm{det}} = W^{-1}W'$. ProDiaL's update rule $W' = s W D_b + \\epsilon$ is a factored stand-in for $W T$: a block-diagonal matrix $D_b$ with block size $r_b$ absorbs the dominant diagonal and block-local rotation, a per-channel scaling vector $s$ adjusts scale, and a low-rank term $\\epsilon = B_\\epsilon A_\\epsilon$ absorbs the residual off-diagonal movement. The learnable parameter count is controlled by $r_b$ and the low-rank value $r_\\epsilon$.","core_discovery":"On its own terms, the paper reports two findings. First, in a Mamba block the components that dominate downstream-task transfer are the Input-Projector and Output-Projector, the linear layers that expand and compress tokens around the SSM, rather than the SSM parameters themselves, even though those parameters directly control the matrices $B$, $C$, and $\\Delta$ in the state-space update. Second, the change from a pretrained projector weight $W$ to a fine-tuned weight $W'$ is well approximated by a linear transformation $W' = W T$ whose matrix $T$ is close to the identity: strong on the diagonal, small off the diagonal, with training gradients concentrated along the diagonal. ProDiaL operationalizes this by freezing $W$ and learning $W' = s W D_b + \\epsilon$, where $D_b$ is a block-diagonal matrix, $s$ is a per-output scaling vector, and $\\epsilon$ is a low-rank correction for the off-diagonal part. With fewer than 1% of the model's parameters learned, the method reports higher or comparable accuracy than full fine-tuning, LoRA, and DoRA on language reasoning tasks and image classification tasks, across Mamba-130M, Mamba-370M, Mamba-1.4B, Vim-tiny, and Vim-small.","pith_inferences":["Extension not in the paper: if the near-diagonal transfer phenomenon holds for large pretrained linear layers generally, a similar diagonal-centric transform could cheapen adaptation of Transformer FFN and MLP blocks; the paper only probes non-attention modules with a single ViT experiment.","Extension: the near-diagonal evidence is visualized for one input projector at one layer on one dataset, so measuring the off-diagonal mass of $T_{\\mathrm{det}}$ across every layer, both projectors, and several tasks would establish how general the mechanism is.","Extension: because $r_b$ and $r_\\epsilon$ form a continuous parameter-accuracy knob, a scheduler that adds off-diagonal capacity only for hard layers or hard datasets could improve cases like Flowers, where the paper reports that more parameters help.","Extension: since only block-diagonal, scaling, and low-rank factors are trained, ProDiaL could be combined with quantized Mamba weights without needing full precision for the frozen projectors, analogous to quantized low-rank recipes for Transformers."],"forward_implications":["PEFT in Mamba should target projectors rather than SSMs; applying existing LoRA and DoRA to projectors outperforms applying the same methods to SSM parameters.","ProDiaL matches or exceeds full fine-tuning and current PEFT baselines on HellaSwag, Winogrande, ARC-E, ARC-C, StanfordCars, Caltech, and Flowers while training less than 1% of the model's parameters.","The method transfers across Mamba 1 and Mamba 2 architectures, across model sizes from 130M to 1.4B parameters, and works when tuning only the input projector or only the output projector.","After training, the transformation can be merged into the frozen projector weights, so no additional parameters need to be stored at inference time.","Ablations show each component contributes: the block-diagonal term alone captures most of the gain, while the off-diagonal low-rank term and the scaling vector add further accuracy."],"supporting_citations":[{"why":"Supplies the 'Strong' SSM-targeted PEFT baseline that the projector-dominance result directly contrasts.","marker":"[17]"},{"why":"Provides the Mamba LLM architecture and pretrained Mamba-130M, Mamba-370M, and Mamba-1.4B weights used in the language experiments.","marker":"[13]"},{"why":"Provides the Vision Mamba architecture and pretrained Vim models used in the vision experiments.","marker":"[53]"},{"why":"Supplies LoRA, both as the low-rank off-diagonal component of ProDiaL's update and as a baseline.","marker":"[20]"},{"why":"Supplies DoRA, the weight-decomposed low-rank baseline ProDiaL is compared against.","marker":"[32]"},{"why":"Supplies BitFit, a bias-only PEFT baseline included in the comparison tables.","marker":"[48]"},{"why":"Provides the Mamba 2 architecture used in the supplementary generalization experiments.","marker":"[5]"}],"fun_headline_variants":["Projectors rule Mamba transfer, not SSMs","ProDiaL: 1% parameter Mamba tuning via diagonal transforms","Mamba's secret: projectors, not state space, dominate learning","Diagonal-centric tuning: tiny Mamba PEFT with big gains","Target projectors only for efficient Mamba fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, across every layer and task, the fine-tuned projector weight is obtained from the pretrained one by a single linear map that is close to the identity matrix; if off-diagonal movement is substantial or the projector is ill-conditioned, the diagonal-centric parameterization cannot represent the needed adaptation.","fun_headline_variants_meta":{"raw":{"variants":["Projectors rule Mamba transfer, not SSMs","ProDiaL: 1% parameter Mamba tuning via diagonal transforms","Mamba's secret: projectors, not state space, dominate learning","Diagonal-centric tuning: tiny Mamba PEFT with big gains","Target projectors only for efficient Mamba fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1895,"prompt_tokens":1006,"completion_tokens":889,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":802}},"tokens_in":622,"tokens_out":889,"duration_ms":8063,"temperature":1.0,"reasoning_tokens":802,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:48:57.829783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $T_{\\mathrm{det}} = W^{-1}W'$ for all projector layers on a held-out task and compare the L1 mass of off-diagonal entries with the diagonal; then train ProDiaL with the off-diagonal term removed. If removing $\\epsilon$ changes accuracy by more than the method's reported gains, or if a synthetic task with a known rotation $W' = W R$, where $R$ is a 45-degree rotation matrix, causes accuracy to collapse, the diagonal-centric assumption is falsified.","supporting_citations":[],"review_version":1}