{"id":"5229eee9-b782-4ef4-87c6-9c21b6046911","arxiv_id":"2412.09875","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SSMI inserts Mamba-based state space modules into LVLMs, fine-tunes only 0.5% of parameters, and reports higher captioning, VQA, and retrieval scores than its baselines.","lead":"The paper proposes fine-tuning large vision-language models by inserting lightweight Mamba state space modules and updating only those parameters. It reports state-of-the-art results on COCO captioning, VQA, and Flickr30k, but provides no code, no hyperparameters, and no base model details, so the claims cannot be checked.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No identity-preserving initialization/residual is specified for the inserted Mamba module (Eq. 1), so the frozen LVLM's forward pass is likely corrupted at initialization, undermining the claim that SSMI preserves pretrained knowledge.","rationale":"The central claim is that SSMI preserves the pretrained LVLM's knowledge while adapting it with only 0.5% trainable parameters. The most load-bearing condition for this claim is the behavior of the model immediately after inserting the Mamba modules but before any training, since the base weights are frozen throughout. The paper's Eq. (1) places the module in series without specifying a residual connection or an identity-initialization scheme; the state-space equations in Eqs. (2)-(3) do not provide one. A randomly initialized Mamba module in series at every layer would change the hidden-state distribution seen by every frozen FFN, so the pretrained knowledge is not preserved at initialization. In addition, the Stage-1 reconstruction objective in Eq. (5) is posed over visual/textual embeddings, not over the MHSA hidden states that the module will see at inference time, so even pretrained modules are not calibrated to their runtime inputs. This is a concrete correctness risk in the method, not merely a missing artifact: it provides a mechanism by which the central premise fails. The reader's verdict of REJECT remains appropriate, since the paper also omits the base LVLM identity, hyperparameters, and artifacts; our concern strengthens the rejection by identifying an architectural reason to doubt the preservation claim. A single implementation-based test with a named public LVLM, comparing zero-shot performance before and after insertion with no fine-tuning, would settle the concern.","tokens_in":8232,"tokens_out":5147,"duration_ms":55128,"concrete_test":"Implement Eq. (1) on a public LVLM (e.g., LLaVA-1.5-7B) with the Mamba modules initialized by the paper's Stage-1 procedure (or default random initialization if unspecified), keep all base weights frozen, and measure the target metric (e.g., COCO Captioning BLEU-4) before any task fine-tuning. Compare with the same LVLM without the Mamba modules (frozen base). If the inserted model's metric collapses toward random or drops by more than 5 BLEU points relative to the frozen base, the insertion disrupts the pretrained representations, directly contradicting the preservation premise. Also report the Mamba initialization scheme and whether a residual connection around the module exists; if no identity initialization or residual is provided, the test is expected to fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines the modified layer as H(l) = FFN(Mamba(MHSA(H(l-1)), V)) (Eq. 1), with the Mamba module placed in series between self-attention and the feed-forward network. No residual connection around the Mamba module and no identity or near-zero initialization are specified; Eqs. (2)-(3) are bare SSM dynamics with learnable A, B, C, D, and the paper never states how these are initialized. If these parameters are randomly initialized and no residual path exists, the Mamba output is a nonlinear random projection of the MHSA output, which is then passed through a pretrained FFN. This distributional shift at every layer means the frozen LVLM's behavior changes immediately at initialization, before any fine-tuning. The claim that freezing most parameters preserves the pretrained model's knowledge therefore has no architectural support in the text. Furthermore, Stage-1 pretraining (Eq. 5) trains Mamba to reconstruct textual embeddings from visual embeddings, but at integration the module receives MHSA hidden states of the target LVLM; these input distributions are not matched, so even the pretrained weights are not calibrated to their runtime inputs. This makes the central preservation premise doubly unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes State Space Memory Integration (SSMI), a parameter-efficient fine-tuning method for large vision-language models (LVLMs). The method inserts lightweight Mamba-based state space modules between the multi-head self-attention (MHSA) and feed-forward network (FFN) sublayers of each transformer layer, freezes the base LVLM, and fine-tunes only the inserted modules. Training proceeds in two stages: first, the Mamba modules are pretrained on a visual-to-text embedding reconstruction loss (Eq. 5); second, they are fine-tuned with a task-specific objective (Eqs. 6-7). Experiments are reported on COCO captioning, VQA v2, and Flickr30k, with the central claim being state-of-the-art performance while updating only 0.5% of the model's parameters (Tables 1-6). The paper also includes ablation studies, a human evaluation, and analyses of robustness, zero-shot transfer, and efficiency.","tokens_in":8369,"tokens_out":4198,"duration_ms":43178,"significance":"If the empirical claims were substantiated, the idea of inserting state space memory modules into frozen LVLMs for parameter-efficient fine-tuning would be a plausible contribution to the PEFT literature, because it adds a new architectural prior distinct from adapters and LoRA. The emphasis on linear-complexity sequence modeling in the visual-language setting is timely. However, as written, the manuscript does not support these claims: the base LVLM is never identified, no training configuration or hyperparameters are given, the results are single point estimates with no error bars, and no code or checkpoints are provided. The architectural description also lacks the initialization or residual design needed to justify the central 'frozen knowledge preservation' premise. The paper is organized and the motivation is clear, but the absence of experimental reproducibility and the incompleteness of the method description make the current claims unverifiable.","major_comments":[{"comment":"The modified layer is defined as H(l) = FFN(Mamba(MHSA(H(l−1)), V)) with no residual connection around the Mamba module and no specification of how A, B, C, D in Eqs. (2)-(3) are initialized. With bare SSM dynamics and random initialization, the Mamba output is a nonlinear random projection of the MHSA output, which changes the hidden-state distribution at every layer before any training. This directly contradicts the claim in §3.3 that freezing most LVLM parameters preserves the pretrained model's knowledge. The authors should either specify an identity or near-zero initialization with a residual connection, or provide measurements (e.g., representation similarity before versus after insertion, or zero-shot performance before any fine-tuning) showing that the frozen backbone is not disrupted.","section":"§3.1, Eq. (1)"},{"comment":"The two-stage training protocol has two load-bearing mismatches. First, the stage-1 objective reconstructs textual embeddings from visual embeddings, but at integration (Eq. 1) the Mamba module receives the MHSA hidden states of the target LVLM at every layer; there is no argument or measurement that these input distributions match, so the pretrained weights are not calibrated to their runtime inputs. Second, Eq. (6) lists CIDEr as an example of a task-specific loss function; CIDEr is a non-differentiable evaluation metric and cannot be directly minimized as a loss without some differentiable surrogate or a reinforcement-learning formulation. These issues undermine the claim that the two-stage procedure provides effective task adaptation.","section":"§3.2, Eqs. (5)-(6)"},{"comment":"No experimental configuration is reported. The base LVLM is never named, and the dataset splits, number of training steps, batch size, learning rate, optimizer, and hardware are all absent. All tables report single point estimates without error bars, confidence intervals, or significance tests, and every baseline (Baseline, Adapter, LoRA, VPT) appears to be self-run with no implementation details. Without this information, the central claim of state-of-the-art performance with 0.5% trainable parameters cannot be assessed or reproduced.","section":"§4, Tables 1-6"},{"comment":"The frequency-domain discretization statement is not verifiable: the text says the system is 'discretized in the frequency domain' and calls z a discretization operator, but Eq. (4) is not derived from Eqs. (2)-(3) and is not a standard formulation of SSM discretization. The authors should either provide a clear derivation (e.g., zero-order hold discretization) or remove the equation, because the model's forward computation is otherwise underspecified.","section":"§3.1, Eq. (4)"}],"minor_comments":[{"comment":"The notation Ŷ is not defined for captioning or retrieval tasks; please specify how the model's prediction is obtained and how the loss is computed for each task.","section":"§3.2, Eq. (6)"},{"comment":"For Flickr30k, the evaluation metrics listed include R@10, but Table 1 only reports R@1 and R@5; please add R@10 or explain its omission.","section":"§4.1"},{"comment":"The human evaluation in Table 3 lacks essential details: the number of annotators, their agreement (e.g., Cohen's kappa), and the exact instruction or rubric used to rate fluency, relevance, and informativeness. Without these, the comparison is not interpretable.","section":"§4.4"},{"comment":"The term 'zero-shot' is used for the adaptability study, but the protocol is not described; please clarify which tasks or datasets were held out during fine-tuning and how the zero-shot evaluation was performed.","section":"§4.5.3"},{"comment":"The reference list contains an unusual number of citations to the same authors (e.g., Zhou and Long 2023a,b,c; Zhou et al. 2024a,b) and several arXiv preprints whose relevance to the specific claims is not explained; please verify that all citations support the statements they are attached to.","section":"References"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as an early draft: the experimental section is a skeleton, the equations are not fully defined, and the architectural description is missing the pieces needed to make the central claim plausible. The reference list is dominated by self-citations and recent arXiv preprints, some of which appear only loosely related to the cited claims; the editorial office may wish to check whether this reflects a standard citation practice. The paper's central claim could in principle be repaired by a major rewriting and new experiments, but as it stands, the empirical evidence is not sufficient for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is not crazy: insert lightweight Mamba modules between self-attention and the feed-forward block in each LVLM layer, freeze everything else, and fine-tune only those modules. If that worked, it would be a useful low-compute adapter, and the paper gets credit for a clean, well-motivated pitch. The writing is organized and the motivation is credible.\n\nBut the supporting evidence is essentially absent. The paper never names the base LVLM, the pretraining corpus, the hyperparameters, or the training configuration. The tables are bare point estimates with no error bars, and every baseline is self-run with no details, so the \"state-of-the-art\" claim is uncheckable. There is no code or data. That alone is a desk-reject in most venues.\n\nWhat pushes this from under-specified to internally inconsistent is the method section. Equation (1) defines H(l) = FFN(Mamba(MHSA(H(l-1)), V)), but Equation (8) defines the same H(l) as just Mamba(MHSA(H(l-1)), V) with no FFN. That is a direct contradiction in the central architecture. Also, no residual connection or initialization is specified for the inserted Mamba module; with random initialization, the frozen model's forward pass would be corrupted at step zero, which contradicts the claim that freezing preserves pretrained knowledge. The Stage-1 pretraining loss reconstructs textual embeddings from visual embeddings, but the module at runtime receives MHSA hidden states, so the input distributions naturally do not match. Using CIDEr as a task loss for captioning is also a red flag, since CIDEr is a metric, not a differentiable objective (unless some surrogate is intended, which is not stated).\n\nThe citation pattern is also thin: several references are to the same author's own work with only loose relevance, and many key claims about adapters or LoRA lack precise attribution.\n\nTo be fair, the modular idea itself is not absurd, and with proper details it might work. But as presented, the paper gives a reviewer nothing to evaluate. I would not send this to peer review in its current form. If the authors were to specify the base model, provide code and error bars, fix the equations, and add a residual or near-identity initialization, it could become a legitimate PEFT contribution worth a second look.","headline":"A coherent PEFT idea that is too under-specified to evaluate: no base model, no training details, and an internal contradiction in the core equation.","tokens_in":9002,"tokens_out":2416,"would_cite":false,"duration_ms":25977,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State space memory modules fine-tune vision-language models with 0.5% of parameters.","keywords":["state space models","Mamba","parameter-efficient fine-tuning","vision-language models","image captioning","visual question answering","text-to-image retrieval","multimodal adaptation"],"falsifier":"Run the SSMI pipeline on COCO Captioning with the inserted state space modules replaced by fixed random projections or identity mappings, keeping the same frozen base model, parameter budget, and training procedure; if the control still reaches the reported BLEU-4 of 38.5, the state space dynamics are not what produces the gain. Conversely, measure a representation-similarity score of the base model's hidden states before and after inserting the modules; if inserting them already degrades the frozen representations, the premise that the base knowledge is preserved fails.","tokens_in":7861,"feed_emoji":"🧠","tokens_out":13452,"duration_ms":117439,"temperature":0.7,"pith_summary":"Large vision-language models are powerful but expensive to fine-tune because most of the model must be updated. This paper tries to show that inserting small state space sequence modules between the attention and feed-forward parts of every layer, and training only those modules while freezing everything else, is enough to adapt the model to new tasks. On COCO Captioning the method reports a BLEU-4 of 38.5 with only about 0.5% of parameters trainable, above both full fine-tuning (36.2) and competing parameter-efficient methods. The same recipe improves visual question answering and text-to-image retrieval, and the paper claims it also gives noise resistance and better zero-shot transfer. If true, this would make adapting large multimodal models practical on small computing budgets.","feed_headline":"0.5% of parameters is enough to adapt a vision-language model","feed_subtitle":"Small inserted memory modules let frozen large models learn new tasks cheaply.","key_machinery":"The load-bearing object is the state space memory module inserted between the multi-head self-attention and the feed-forward network in every layer of the LVLM. It is a selective state space model—a recurrence with input-dependent matrices $A$, $B$, $C$, and $D$—which processes sequences in linear time and encodes long-range dependencies in its hidden state. The mechanism works in two stages: a pretraining stage that aligns the modules to vision-language data by reconstructing text embeddings from visual embeddings, and a task-specific fine-tuning stage that adjusts only the inserted weights while all base LVLM weights stay frozen. Equation (1) makes this insertion the entire architectural change, and Eq. (10) states that the trainable-to-total parameter ratio is much smaller than one. The claimed gain comes from this narrow, trainable memory pathway through an otherwise frozen model.","core_discovery":"The central claim is that inserting lightweight state space modules—called SSMI—into a frozen large vision-language model and fine-tuning only those modules captures the long-range visual and sequential patterns a task needs without disturbing the pretrained representations. The updated hidden state at layer $l$ is written as $$$H^{{(l)}}$ = \\mathrm{FFN}(\\mathrm{Mamba}(\\mathrm{MHSA}($H^{{(l-1)}}$), V))$$, where Mamba is a linear-time selective state space layer with dynamics $s_{t+1} = A s_t + B h_t$ and $y_t = C s_t + D h_t$. The modules are first pretrained to reconstruct textual embeddings from visual embeddings, then fine-tuned on the target task with a combined loss. Because the base LVLM is frozen, the trainable parameter fraction is tiny: the paper reports roughly 0.5%, with COCO captioning BLEU-4 at 38.5, VQA accuracy at 71.2, and Flickr30k R@1 at 64.2, all above the baselines compared in the paper.","pith_inferences":["A natural extension the paper does not test is whether the same inserted modules work for other frozen sequence models, such as pure language models or video-language models, since the insertion point is not specific to vision.","The pretraining stage's paired visual-textual dataset is not specified, so the total compute cost of SSMI including pretraining may be larger than the 0.5% parameter number suggests; a fair comparison would report end-to-end training cost versus low-rank adapters.","If the frozen-representations premise holds, the method suggests a modular view of adaptation: a small trainable memory layer can inject task-specific behavior into a fixed pretrained network, which would also open a route to swapping tasks by swapping modules.","The reported gains are on standard benchmark splits; testing on out-of-distribution images or on tasks requiring precise spatial reasoning would clarify what long-range visual dependencies the modules actually capture."],"forward_implications":["COCO captioning BLEU-4 rises to 38.5 while training roughly 0.5% of parameters, so large-model adaptation can run without full-model gradients.","VQA accuracy reaches 71.2 and Flickr30k R@1 reaches 64.2 with the same recipe, so the inserted modules transfer across captioning, question answering, and retrieval tasks.","Removing either the state dynamics or the visual embedding integration drops BLEU-4 from 38.5 to 37.2 or 36.5, so both components are claimed to contribute to the result.","Under synthetic input noise and in zero-shot settings, SSMI degrades less than the compared methods, so the efficiency gain is claimed to come with resilience."],"supporting_citations":[{"why":"Supplies the Mamba selective state space model that SSMI inserts; the paper's efficiency claim rests on its linear-time and low-memory sequence modeling.","marker":"[Gu and Dao, 2023]"},{"why":"Defines the adapter-family baseline and the limitation that SSMI claims to overcome: scalability and weak long-range dependency capture.","marker":"[Hu et al., 2023]"},{"why":"Motivates parameter-efficient fine-tuning as the design goal and situates the two-stage training strategy within that line of work.","marker":"[Zhong et al., 2024]"}],"fun_headline_variants":["Frozen vision-language models learn new tasks with 0.5% parameters unfrozen","Tiny state space modules make large vision-language fine-tuning cheap","Adapt LVLMs with 0.5% trainable parameters via Mamba-based memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that inserting small trainable modules into every layer of a frozen model leaves the pretrained representations intact, and that pretraining those modules to reconstruct text embeddings from visual embeddings transfers to the downstream task; the paper does not directly measure either.","fun_headline_variants_meta":{"raw":{"variants":["Frozen vision-language models learn new tasks with 0.5% parameters unfrozen","Tiny state space modules make large vision-language fine-tuning cheap","Adapt LVLMs with 0.5% trainable parameters via Mamba-based memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000973,"raw_usage":{"total_tokens":4126,"prompt_tokens":928,"completion_tokens":3198,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3131}},"tokens_in":544,"tokens_out":3198,"duration_ms":27598,"temperature":1.0,"reasoning_tokens":3131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:37:50.135316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the SSMI pipeline on COCO Captioning with the inserted state space modules replaced by fixed random projections or identity mappings, keeping the same frozen base model, parameter budget, and training procedure; if the control still reaches the reported BLEU-4 of 38.5, the state space dynamics are not what produces the gain. Conversely, measure a representation-similarity score of the base model's hidden states before and after inserting the modules; if inserting them already degrades the frozen representations, the premise that the base knowledge is preserved fails.","supporting_citations":[],"review_version":1}