{"id":"98ef6595-ebed-4ba7-8c19-3ddb714f71c2","arxiv_id":"2501.02040","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VMINet, a backbone built entirely from Mamba-inspired separable self-attention blocks with a position-encoding triangular mask, matches or beats Vision Mamba on classification and dense prediction benchmarks.","lead":"This paper introduces VMI-SA, a self-attention variant that borrows design ideas from Mamba state-space models, including depthwise local convolution and a masked context vector that keeps computational cost linear. The authors build a demonstrative backbone named VMINet and report ImageNet, COCO, and ADE20K results that are competitive with Vision Mamba and several lightweight backbones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rank-based justification for the causal mask (Eq. 8) is likely vacuous in the trained regime: the mask changes positional order, not rank; the paper's own ablation attributes the gain to positional information.","rationale":"The paper's central empirical claim is credible: VMINet variants beat Vim at comparable parameters, and the ablations show both the attention branch and the mask contribute. The weakest point is not the experiments but the theory that explains the mask. The reader flagged the unproven premises in Eq. (8). I want to stress-test further: in a trained net, Q and K come from learned linear maps applied to a feature map, so the row-scaled matrix softmax(Q)⊙K is generically full column rank already; if so, the causal mask cannot increase its rank, and the derivation in Eq. (8) is vacuous for the actual model. Even if the mask does increase rank in some layers, Eq. (11) shows the concrete effect is a fixed positional weighting (later tokens contribute more often), and the paper explicitly attributes the benefit to loss of positional information rather than to rank. Table 4's different masks changing accuracy reinforces that the specific ordering pattern matters. Thus the theoretical support for the central novel component is a post hoc narrative. This does not destroy the empirical contribution, but it means the paper should either establish the rank mechanism empirically or reframe the mask as a learned positional bias and remove the 'theoretically increase diversity' claim. The reader's CONDITIONAL verdict remains appropriate; no change needed.","tokens_in":12175,"tokens_out":11122,"duration_ms":124476,"concrete_test":"On the trained no-mask VMINet-XS baseline, extract Q and K at each stage and compute the numerical rank (SVD, threshold ~1e-5 of the largest singular value) of softmax(Q)⊙K. If the rank is already min(L,D) before applying M, Eq. (8)'s premise fails and the gain from M must be due to the order bias in Eq. (11), not to rank. As a second check, retrain VMINet-XS with an order-reversed (upper-triangular) mask, keeping sparsity and the α/β scheme identical; if Top-1 changes by more than ~0.5, the mask's benefit is order-specific, not rank-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing theoretical step is Section 3.3.2, Eq. (8): for an all-nonzero L×D matrix A, zeroing the upper triangle maximizes rank to D, and the paper concludes that 'the introduction of causality into the separable self-attention can theoretically increase the diversity of contextual information, thereby enhancing performance.' This conclusion only has force if the unmasked matrix softmax(Q)⊙K is rank-deficient. But K is produced by a learned linear map on a high-dimensional feature map; nothing prevents it from being full column rank generically, so the mask cannot 'increase' rank. The algebra in Eq. (11) shows what the mask actually does: it changes which tokens are summed into each entry of the context vector, with higher-index tokens encoded more frequently. That is a positional/order bias, not a rank increase. The paper's own ablation section makes this explicit: removing M 'leads to a significant performance degradation ... primarily due to the loss of positional information.' The claimed rank-to-performance link is therefore not merely unproven; it is not the mechanism behind the observed gain, and the causal-mask design rests on a post hoc story. The empirical headline may still be true, but the stated theoretical derivation should not be accepted as the explanation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VMI-SA, a linear-complexity separable self-attention variant that adds a depthwise-convolution branch, trainable token weights, and a lower-triangular/banded mask, together with a local residual term. It derives a recurrent form and a parallel matrix form, then builds VMINet by stacking VMI-SA blocks with simple downsampling. Experiments on ImageNet-1K, COCO object detection/instance segmentation, and ADE20K semantic segmentation report competitive results and consistent gains over Vim, with ablations isolating the contributions of the attention operation and different mask matrices. Code is publicly released.","tokens_in":12349,"tokens_out":10350,"duration_ms":107535,"significance":"The empirical contribution is meaningful if reproducible: VMINet is a simple, lightweight backbone whose ablations clearly show that the attention-like global term and the mask both matter (removing attention costs 2.2 points, removing the mask costs 2.1 points). The algebraic expansion in Eqs. (4)-(6) is correct, and the mask-form ablation in Table 4 is informative. The main weakness is the theoretical motivation: the rank-increase argument in Section 3.3.2 is not a valid derivation of the mask's benefit in the trained regime, and the recurrent-form indexing does not match standard causal sequence modeling. These issues affect the paper's central conceptual claim, but they are reparable by reframing or by adding measurements.","major_comments":[{"comment":"The statement that introducing causality 'can theoretically increase the diversity of contextual information, thereby enhancing performance' is not established by the rank argument. For A = softmax(Q) ⊙ K with K ∈ R^{L,D}, one already has rank(A) ≤ D from Eq. (7); if K is full column rank, which is the generic case in a trained network, rank(A) = D before masking, so M cannot increase the rank to min{L,D} = D. The all-nonzero premise is unverified, and in the actual VMI-SA of Eq. (10) Q is not a scalar softmax score, so the formal analogy is loose. Eq. (11) shows what M actually does: it changes which tokens are summed into each context entry, with higher-index tokens encoded more frequently, and Section 4.2 attributes the gain to loss of positional information. Furthermore, Table 4 is inconsistent with a pure rank mechanism: banded and lower-triangular masks can both have full column rank D, yet the hybrid mask gives a further gain. Please remove the rank-increase claim and describe the mask as a positional/order bias, or supply direct evidence such as measured ranks of Q⊙K before and after masking and an experiment that varies rank while controlling position.","section":"3.3.2, Eq. (8)"},{"comment":"The recurrent form does not implement the stated 'restrict the receptive field to the previous token' in the usual sequence sense. In Eq. (9), h_i accumulates Q_i ⊙ K_i over tokens, but M_i is the i-th row of the L×D matrix M, so M_i ⊙ h_i gates feature dimensions according to the row index i rather than limiting h_i to the previous token. Eq. (11) confirms this reading by expressing e_n as a sum over t ≥ n, meaning the mask is lower-triangular in the (token, feature-dimension) plane and not a causal mask over the token sequence. The authors should either define M with a proper token-causal indexing or revise the recurrent-form discussion so that the analogy to Eq. (3) is not overstated.","section":"3.3.3, Eq. (9)"}],"minor_comments":[{"comment":"The text says element-wise multiplication projects the feature vector into a 'C^2 dimensional' space, but after collecting symmetric terms the number of distinct monomials is C(C+1)/2; please correct the dimension statement.","section":"3.1, Eq. (4)"},{"comment":"The notation for β is inconsistent: Eq. (9) uses β_i per token while Eq. (10) uses a scalar β with no index; please specify whether β is per-token or shared.","section":"3.3.3, Eq. (9) and Eq. (10)"},{"comment":"The matrix M is stated to be in R^{L,D} but the displayed matrix appears square; for L > D, please clarify which D columns of the lower-triangular pattern are used.","section":"3.3.2, Eq. (8)"},{"comment":"The claim 'for the first time introducing some excellent design concepts of Mamba into separable self-attention' is a strong novelty statement that is hard to verify; consider softening it.","section":"Abstract and Introduction"},{"comment":"The assertion that the attention information in softmax(Q) ⊙ K is 'severely homogenized' is not quantified; please provide a concrete measure or remove the adjective.","section":"3.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical package is honest and the code release is a plus, but the theoretical narrative in Section 3.3.2 will likely draw strong criticism from reviewers. The rank argument should either be backed by measured rank statistics and controlled experiments or replaced by a positional-information explanation. If the authors make this revision carefully, the paper could become acceptable; the current version overstates the derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on VMINet: the empirical package is the real content, and it's mostly solid. What's new is the specific design — separable self-attention with a depthwise-conv branch, a triangular/banded mask, and trainable token weights — plus a clean prototype backbone that reports competitive numbers against Mamba at similar size. The ablations are genuinely informative: removing the attention branch costs ~2 points, removing the mask costs ~2 points, and the mask-form comparison (banded vs triangular vs block-diagonal) shows the mask is doing real work. The algebraic expansion in Eqs (4)-(6) is correct, and the code is released.\n\nThe soft spot is the theory section, and it's not minor. Eq (8) claims a causal mask maximizes rank of the context matrix under an \"all elements nonzero\" assumption. That assumption is not guaranteed in trained models, and more importantly, the mask doesn't actually increase rank in the trained regime — the unmasked Q⊙K is generically full column rank, so zeroing the upper triangle changes which tokens get summed, not the rank. Eq (11) shows exactly that: higher-index tokens get encoded more often. That's a positional/order bias, not a rank increase. The paper's own ablation table says the gain is \"primarily due to the loss of positional information.\" So the rank-to-performance derivation is a post hoc story, and the authors should either soften the claim or replace it with a positional-encoding argument.\n\nThat said, the central empirical claim — the module and backbone work — holds up as far as I can tell. Single-run results without confidence intervals are the norm in this subfield, so I won't dock much for that. The comparisons against Vim and VMamba look fair, and the parameter-efficiency story is compelling.\n\nBottom line: the theory overreaches, but the design and experiments justify a serious look. Worth sending to peer review. I'd probably cite it as an instance of Mamba-inspired linear attention, with a note about the weak rank argument.","headline":"Solid empirical contribution with a shaky theoretical motivation: the mask works, likely for positional reasons, not the rank argument in Eq. (8).","tokens_in":12996,"tokens_out":1845,"would_cite":true,"duration_ms":18632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VMINet shows that a Mamba-inspired separable self-attention with a causal mask beats Vision Mamba at linear cost.","keywords":["separable self-attention","state space model","Vision Mamba","linear complexity attention","causal mask","depthwise convolution","image classification","VMINet"],"falsifier":"Train VMINet-XS with the mask removed but with learned positional encodings added to tokens; if that model matches or exceeds the masked model's 78.6 top-1 accuracy, the causal mask's benefit is positional, not rank-based.","tokens_in":1609,"feed_emoji":"🖼️","tokens_out":2079,"duration_ms":71692,"temperature":0.7,"pith_summary":"This paper tries to show that the design ideas behind Mamba, a linear-complexity state space model, can be transplanted into separable self-attention without paying quadratic cost. It proposes VMI-SA, which combines a depthwise-convolution local branch, a causal, banded, or block-diagonal mask over token interactions, and trainable per-token weights, and builds VMINet by stacking this module with simple downsampling layers. The reported result is that VMINet consistently outperforms Vision Mamba (Vim) on ImageNet classification, COCO detection, and ADE20K segmentation at comparable or lower parameter counts, and stays competitive with strong CNN and Transformer backbones. If the claim holds, it points to a route to linear-complexity global modeling that avoids Mamba's causal scanning and specialized SSM kernels.","feed_headline":"Causal-mask attention beats Vision Mamba at linear cost","feed_subtitle":"A 2–28M-parameter network using VMI-SA edges out Mamba-based models on classification and dense prediction.","key_machinery":"The central object is the sparse mask matrix $M$ applied to the elementwise token product $Q \\odot K$ before summation into a context vector. Equation (8) shows that zeroing the upper triangle of a nonzero matrix yields rank $\\min\\{L,D\\}$, and the paper treats this maximum rank as richer contextual information. Around this, the module has a depthwise-convolution branch for local correlation, learnable scalars $\\alpha_i$ controlling token importance and $\\beta_i$ balancing local versus global information, and two equivalent forms: a recurrent form restricted to previous tokens and a parallelizable matrix form with a global receptive field. The matrix form is what VMINet stacks.","core_discovery":"For any image sequence, VMI-SA computes a global context vector by elementwise-multiplying token-wise query/key features, weighting each token with learnable scalars, applying a sparse mask (lower-triangular, banded, block-diagonal, or hybrid), and broadcast-multiplying the result across values. The paper argues that elementwise multiplication already maps features into a higher-dimensional nonlinear space, that the mask raises the rank of the attention information to $\\min\\{L,D\\}$, and that restoring the global receptive field in matrix form keeps the complexity linear. Its central empirical discovery is that a plain VMINet with 2.0–28.4M parameters outperforms Vim at matching scales: 78.6 versus 76.1 top-1 for the small variant, and 82.4 for VMINet-B against 82.2 for VMamba-T, with similar gains on COCO and ADE20K.","pith_inferences":["If adding learned positional encodings without any mask recovers the roughly two-point gap between masked and unmasked VMINet, the rank-maximization story would be a post-hoc explanation rather than the mechanism; the ablation data alone do not separate rank from position.","The fact that a banded matrix with bandwidth $B/2$ matches the triangular matrix in accuracy suggests the practical benefit is mostly local and global position bias, not literal maximal rank, because the banded matrix has lower rank.","Because the module is a drop-in attention layer using only depthwise convolution, elementwise products, and masks, a direct stress test would be to vary the mask bandwidth continuously from zero to full length and record accuracy; if accuracy saturates before full triangular context, the 'maximal rank' design point is not the operating one."],"forward_implications":["Linear-complexity global modeling can be applied to high-resolution dense prediction without quadratic attention cost or Mamba's selective-scan mechanism.","Images need not be flattened into 1D sequences: the mask and context vector act on 2D features, preserving spatial correlation while keeping the same complexity.","The exact triangular causal mask is not essential: banded and hybrid masks match or exceed it, so the benefit is a tunable positional and importance bias rather than a uniquely causal one.","The recurrent form, being an RNN-like linear attention, provides a natural route to causal sequence modeling outside images, as the paper notes for other fields.","The approach transfers well to smaller models: VMINet-XS reaches 78.6 top-1 with 7M parameters and 1.4 GFLOPs, making it attractive for efficient vision backbones."],"supporting_citations":[{"why":"Defines the separable self-attention baseline (scalar query, context vector, broadcast multiplication) that VMI-SA starts from and improves.","marker":"[16]"},{"why":"Provides the Mamba/SSM design, including the local convolution branch, no softmax, and recurrent hidden state, which VMI-SA imports.","marker":"[3]"},{"why":"Supplies the Vim baseline, block configuration, training recipe, and main performance comparison target.","marker":"[28]"},{"why":"Supports the causal linear-attention setting, the forget gate, and the local-to-global receptive field observation that motivate the recurrent form.","marker":"[5]"},{"why":"Cited for the claim that softmax attention matrices are usually full rank, which frames the rank-enhancement motivation.","marker":"[4]"},{"why":"Unifies state space models and structured attention, providing the theoretical bridge the paper relies on.","marker":"[2]"}],"fun_headline_variants":["Mamba-inspired separable attention outperforms Vision Mamba","Separable attention with linear complexity beats Vim","VMI-SA: Mamba-style attention for vision","Linear-complexity attention edges out Vision Mamba"],"cache_read_input_tokens":14976,"weakest_assumption_plain":"The argument that the causal mask helps because it maximizes the rank of the attention information assumes the trained token products are all nonzero and that higher rank, rather than positional information, is what actually boosts accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Mamba-inspired separable attention outperforms Vision Mamba","Separable attention with linear complexity beats Vim","VMI-SA: Mamba-style attention for vision","Linear-complexity attention edges out Vision Mamba"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0004,"raw_usage":{"total_tokens":2064,"prompt_tokens":898,"completion_tokens":1166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1104}},"tokens_in":514,"tokens_out":1166,"duration_ms":9647,"temperature":1.0,"reasoning_tokens":1104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:19:57.819528+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train VMINet-XS with the mask removed but with learned positional encodings added to tokens; if that model matches or exceeds the masked model's 78.6 top-1 accuracy, the causal mask's benefit is positional, not rank-based.","supporting_citations":[{"cited_title":"Mehta and M","cited_arxiv_id":null,"evidence_quote":"Defines the separable self-attention baseline (scalar query, context vector, broadcast multiplication) that VMI-SA starts from and improves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Vim baseline, block configuration, training recipe, and main performance comparison target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the causal linear-attention setting, the forget gate, and the local-to-global receptive field observation that motivate the recurrent form."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited for the claim that softmax attention matrices are usually full rank, which frames the rank-enhancement motivation."}],"review_version":1}