{"id":"2edcbd52-e1e5-4cff-8e88-be11d9966b5c","arxiv_id":"2412.08947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Selective Visual Prompting generates input-dependent token prompts inside every Vision Mamba layer, outperforming prior visual prompting methods on the HTA and VTAB-1K benchmarks.","lead":"This paper introduces Selective Visual Prompting (SVP), a way to adapt pre-trained Vision Mamba models to new image tasks by adding small token-level prompts to every layer instead of prefixing prompts at the start. It reports consistent accuracy gains over prior prompt-tuning methods on standard benchmarks with few trainable parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No parameter-matched non-selective control: SVP's per-layer prompt generators add ~1.6M tunable params, so the reported gains may reflect added capacity rather than input-dependent gate selectivity.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing gap: SVP's design couples input-dependence with a large increase in tunable parameters, and no experiment separates these two factors. The paper's own claims lean heavily on the word 'selective' — the abstract, introduction, and Section 3 all say that token-wise, input-dependent prompts activate the update/forget gates in a way static prompt tokens cannot. If a parameter-matched non-selective control performs equally well, the mechanism would be refuted even though the method might still be a useful PEFT recipe. This is a genuine correctness risk for the paper's central explanatory claim, not merely a disagreement with the field's preferred baselines. The authors do provide real supporting evidence: the method is evaluated on two standard benchmarks, ablations remove components, code is released, and the comparisons show consistent improvements. Those facts make the paper valuable but do not settle the selectivity-versus-capacity question. The requested control is cheap and decisive, so the appropriate disposition remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":14362,"tokens_out":4331,"duration_ms":49063,"concrete_test":"On HTA (minimum: CIFAR100, DTD, CUB, Flowers), train a matched non-selective control: use the exact SVP architecture and optimizer, but replace the input to G^C and G^I with one fixed per-layer learnable vector z_i (same shape as x_i) instead of x_i, keeping the layer structure and parameter count essentially identical. Run SVP and this static-prompt control with 3 seeds and report mean±std for both. If the static control matches SVP within about 1 point on average, the 'selective/input-dependent' mechanism is unsupported; if SVP exceeds the control by more than 3 points, selectivity is a genuine driver of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equations 4–7 define SVP as per-layer generators G^C and G^I that map each token x_i to a token-wise prompt, producing \\bar p_i = α⊙p^C_i + β⊙p^I_i, which is added to x_i before each Mamba block. The central mechanistic claim is that input-dependent selectivity—not raw capacity—drives the large gains over VPT and DAM-VP on Vim-S. The experiments never isolate this factor. The reported tunable budget is ~1.6M parameters; VPT-deep on Vim-S uses only about 24×384≈9K prompt-token parameters, so SVP has roughly 170× more learnable parameters. Table 4 ablates whole components (IP/CP) rather than holding capacity fixed while removing input-dependence, and Table 3 only varies the position of appended prompts. Thus the observed gains could be explained by the added per-layer low-rank transform (which is effectively what G^I is) even if the input-dependent 'selective' path contributed nothing. The gate visualizations in Figures 6–7 show activation differences but do not include a static-prompt control, so they cannot distinguish the selectivity hypothesis from the capacity hypothesis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Selective Visual Prompting (SVP), a parameter-efficient fine-tuning method tailored to Vision Mamba (Vim). Instead of appending a fixed prompt sequence, SVP adds lightweight per-layer generators that produce token-wise prompts from the input image tokens: a shared Cross-Prompting generator for cross-layer information and per-layer Inner-Prompting generators for layer-specific information, combined through learnable element-wise scaling factors. The prompted tokens are fed into frozen Vim blocks, with only the prompt generators, scaling factors, and classification head trained. The authors evaluate on HTA (10 datasets) and VTAB-1K (19 datasets) with a Vim-Small backbone, reporting average accuracies of 89.8% on HTA and 70.1% on VTAB-1K, surpassing VPT, DAM-VP, SPT, and AutoVP baselines. They also provide ablations of prompt format/position, component contributions, hyperparameters, and visualizations of update gate activations. The paper claims this is the first visual prompting method specifically designed for Vim and attributes the gains to input-dependent selectivity that activates Vim's update and forget gates.","tokens_in":14616,"tokens_out":5005,"duration_ms":53253,"significance":"If the reported results hold, SVP would be a useful and practical PEFT method for Mamba-style vision backbones, and the paper would be one of the first to explicitly address the sequential token-wise processing of Vim in a prompting framework. Strengths include broad benchmark coverage on HTA and VTAB-1K, a publicly released codebase, ablations of the main design choices, and gate-level visualizations that connect the method to the internal mechanism of Mamba. The main weakness is that the central mechanistic claim—that input-dependent selectivity, rather than added per-layer capacity, drives the improvements—is not isolated by any parameter-matched control. Because the proposed generators are essentially per-layer low-rank transformations, the reported gains could in principle be reproduced by a non-selective per-layer adapter or static prompt generator with the same parameter budget. The theoretical discussion in Equations (11)–(12) is substitutional and does not by itself prove the selectivity mechanism.","major_comments":[{"comment":"The central claim that input-dependent selectivity drives the gains is not isolated from added parameter capacity. SVP adds per-layer Inner-Prompting generators (linear down/up with SiLU) plus a shared Cross-Prompting generator, totaling roughly 1.6M tunable parameters, whereas the VPT-deep baseline on Vim-S adds only prompt-token parameters on the order of 9K. The ablations in Table 4 remove entire components, so they vary input-dependence and capacity simultaneously. I request a parameter-matched control: for example, a non-selective variant in which each layer's prompt is a single learned vector, or a per-layer low-rank adapter on x_i with the same rank and parameter budget that is not a function of the input. Without such a control, the improved accuracy in Tables 1 and 2 could be explained by the additional per-layer low-rank transform alone, which would undermine the paper's mechanistic interpretation.","section":"§3, Eqs. (4)–(7); Tables 1–2; Table 4"},{"comment":"The theoretical account in Equations (11)–(12) is a substitution: it rewrites the Mamba recurrence after replacing x_i with x_i + \\bar p_i. It shows that the prompted input appears in the input-dependent B_i, C_i, and \\Delta_i maps, but it does not establish that this promotes discriminative information propagation, nor does it distinguish selectivity from any other input perturbation. Since B_i, C_i, and \\Delta_i are input-dependent by construction, the statement that prompts activate the gates is partly tautological and not falsifiable as stated. Please either provide a precise statement—for example, a condition on gate magnitudes or hidden-state norms—or reframe the discussion as an empirical observation supported by controlled experiments rather than a derivation.","section":"§4, Eqs. (11)–(12) and Discussion"},{"comment":"The number of shared layers in Cross-Prompting is reported as \"set to 4, 8, or 12, depending on the dataset,\" but no selection rule is described. If this hyperparameter is tuned per dataset using the test set, the reported averages may overstate generalization. Please specify whether the choice is made on a validation split and with what criterion, report the chosen value for each of the 29 datasets, and show the sensitivity of the final averages to this choice. The same issue applies to the hidden dimension of the Inner-Prompting generator, which is set to 64 after observing the trend in Figure 5; a selection rule should be stated.","section":"Implementation Details and Figure 4"},{"comment":"All accuracies are reported without error bars or number of seeds. On several datasets the differences are small (for example, SVP vs. DAM-VP on GTSRB is 97.5 vs. 96.5, and on CIFAR10 it is 98.6 vs. 98.0), and single-run results are insufficient to support the claim of significant improvement. Please report mean and standard deviation over at least three independent runs for the main comparisons, or otherwise provide a variance estimate.","section":"Tables 1 and 2"},{"comment":"The VTAB-1K comparison includes only VPT and SPT among prompting baselines, while the HTA table includes DAM-VP, E2VPT, AutoVP, and SA2VP. Since the abstract and conclusion claim superiority over state-of-the-art visual prompting methods, the VTAB-1K evaluation should either include the same set of baselines or explicitly state why they are omitted. Without this, the breadth of the SOTA claim is not fully supported.","section":"Table 2"}],"minor_comments":[{"comment":"The word \"pracatically\" should be \"practically\".","section":"§4, paragraph before Eq. (11)"},{"comment":"The word \"sequencial\" should be \"sequential\".","section":"Ablation Study, 'Ablation of Prompt Format and Position'"},{"comment":"The manuscript alternates between \"update and output gates\" and \"update and forget gates\"; please use consistent terminology, preferably matching the Mamba formulation used in the paper.","section":"Abstract and Introduction"},{"comment":"The text inside these figures appears corrupted or poorly rendered (e.g., unicode fragments), and the axes are not labeled clearly. Please provide clean vector figures with readable legend and axis labels.","section":"Figures 4 and 5"},{"comment":"The update-gate visualizations show differences between methods, but without a static-prompt or parameter-matched control they cannot distinguish the selectivity mechanism from added capacity; if these figures are meant to support the mechanism, a quantitative summary (e.g., mean gate values over layers) and the requested control would strengthen the interpretation.","section":"Figures 6 and 7"},{"comment":"The paper does not report the computational overhead of generating token-wise prompts at every layer; adding a sentence on latency or FLOPs relative to the frozen backbone would help practitioners assess the efficiency claim.","section":"Overall"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and useful problem, and the proposed method is simple and likely reproducible given the released code. The main risk is that the title-level claim about selectivity is not currently supported by a parameter-matched control; this is fixable but requires additional experiments. I would also encourage the authors to tighten the hyperparameter-selection reporting before publication, since the per-dataset choice of shared-layer count is central to the reported averages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SVP is a solid empirical paper. It is the first to design token-wise, input-dependent prompts specifically for Vision Mamba, and the reported gains over VPT, DAM-VP, and SPT on both HTA and VTAB-1K are large and consistent. The paper is worth a serious referee, but the reviewer should push on one load-bearing point: the mechanism is not isolated from raw parameter capacity.\n\nWhat is new and good. The prompt design is genuinely tailored to the Mamba architecture. Instead of appending a few fixed prompt tokens, SVP runs a lightweight generator per token at each layer, producing an input-dependent perturbation before the Mamba block. The dual-path split into shared cross-layer and per-layer inner generators is sensible, and the ablations show both components help. The empirical work is broad: 10 HTA datasets plus 19 VTAB-1K tasks, with a frozen Vim-S backbone and about 1.6M tunable parameters. The code is available. That is real engineering and evaluation effort, and the gains are too consistent to wave away.\n\nWhere it is soft. The main weakness is exactly what the stress-test says. VPT on Vim-S uses roughly 9K prompt-token parameters; SVP adds ~1.6M via per-layer low-rank generators. The ablations in Table 4 remove entire components, so they cannot separate the contribution of input-dependence from the contribution of added capacity. A static per-layer prompt, or a per-layer adapter with the same parameter budget, is the missing control. The gate visualizations in Figures 6-7 show activation differences, but they lack a static-prompt baseline, so they do not discriminate between the selectivity hypothesis and a simple capacity effect. The theoretical account (Equations 11-12) is essentially a rewrite of the recurrence after substituting prompted inputs; it does not derive a prediction or a bound, and the statement that prompts activate gates is true by construction. I would not call it circular, just not a proof. Also, the number of shared layers in Cross-Prompting is tuned per dataset (4, 8, or 12) without a stated selection rule, and there are no error bars anywhere. VTAB-1K is missing some baselines (DAM-VP, AutoVP) that appear in the HTA table, which makes the comparison uneven across benchmarks.\n\nVerdict. The method works at a practical level; the mechanism is a reasonable hypothesis but not established. The missing control is addressable—one parameter-matched static-prompt or adapter baseline would go a long way. I would encourage a careful referee to request that, along with a hyperparameter selection rule and ideally some variance estimates. This is not a desk reject. It is a useful contribution to the PEFT-for-Mamba subfield, and it deserves reviewer time.","headline":"SVP is a solid empirical win for prompting Vision Mamba, but the selectivity mechanism is not isolated from added parameter capacity; worth a real review.","tokens_in":15132,"tokens_out":3187,"would_cite":true,"duration_ms":30581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token-wise selective prompts adapt frozen Vision Mamba better than prefix prompts.","keywords":["Selective Visual Prompting","Vision Mamba","state space models","parameter-efficient fine-tuning","visual prompt tuning","update and forget gates","HTA benchmark","VTAB-1K"],"falsifier":"Build a non-selective version of SVP with the same parameter budget and dual-path structure in which each prompt is a learned constant per layer (or per layer group) rather than a function of the input, and compare on HTA and VTAB-1K; if it matches SVP's accuracy, the claim that input-dependent gate activation is the cause is refuted.","tokens_in":14165,"feed_emoji":"🖼️","tokens_out":6809,"duration_ms":60968,"temperature":0.7,"pith_summary":"The paper sets out to show that the usual recipe for visual prompting—prepending a few learned tokens to the image sequence—is the wrong interface for Vision Mamba, because Mamba compresses information token by token through input-dependent gates. Its proposed method, Selective Visual Prompting (SVP), instead generates a prompt for every image token at every layer from that token itself, using two lightweight generators: one shared across layers and one per layer. These prompts are added to the tokens before they enter Mamba blocks, so the update and forget gates are computed from prompted inputs and can be steered to keep discriminative information. If the claim is right, a frozen Vim-Small backbone with roughly 1.6M tunable parameters reaches 89.8% average accuracy on HTA and 70.1% on VTAB-1K, beating prefix-prompt and image-prompt baselines. That would make Mamba-style backbones practically adaptable without full fine-tuning.","feed_headline":"Per-token selective prompts lift frozen Vision Mamba to 89.8%","feed_subtitle":"SVP tunes only 1.6M parameters, yet beats prior visual prompting on HTA and VTAB-1K.","key_machinery":"The load-bearing object is the selective prompter, a lightweight generator attached to each Mamba block. Cross-Prompting uses one fully connected generator with parameters shared across a group of layers (4, 8, or 12) to produce prompts that carry common information; Inner-Prompting uses a per-layer bottleneck MLP—linear down to 64 dimensions, SiLU activation, linear up—to produce layer-specific prompts. Two zero-initialized elementwise scaling vectors, $\\alpha$ and $\\beta$, learn how much of each path to add, and the combined prompt is added directly to every image token before the block. This machinery matters because it changes the inputs to the functions $S_B$, $S_C$, and $S_\\Delta$ that generate Mamba's data-dependent gates, so the prompt controls, at every token, which information is updated into and forgotten from the recurrent state.","core_discovery":"On the paper's own terms, the central discovery is that input-dependent, token-wise prompts can activate the selective gates of Vision Mamba in a way that prefixed prompt tokens cannot. For each token $x_i$, SVP forms $\\bar{p}_i = \\alpha \\odot p^C_i + \\beta \\odot p^I_i$ from a cross-layer prompt $p^C_i$ and an inner-layer prompt $p^I_i$, then feeds $x_i + \\bar{p}_i$ into the Mamba block. Because Mamba computes its input-dependent parameters $B_i$, $C_i$, and $\\Delta_i$ from the prompted token, the recurrence's update term $B_i(\\Delta_i \\odot x^p_i)$ and forget factor $\\exp(S_\\Delta(x^p_i) \\odot A)$ are directly affected by the prompt at every sequence position. The paper argues that this selective activation is what lets the model write discriminative features into the hidden state and discard distractions, and it supports the argument with accuracy gains on two benchmarks and with visualizations of update-gate activity across layers.","pith_inferences":["Because the gating mechanism SVP exploits is generic to selective state-space models, the same prompt design should transfer to larger or bidirectional Mamba vision backbones beyond the Vim-Small tested here; this is an extrapolation, not a result in the paper.","A parameter-matched control with input-independent per-layer prompts would separate the contribution of selectivity from the contribution of added capacity; without that control, the mechanistic story is not fully isolated.","Since $\\alpha$ and $\\beta$ start at zero, SVP preserves the pre-trained model's output at initialization, which suggests it could be composed with other parameter-efficient adapters such as LoRA without disturbing the base model's behavior.","The update-gate visualizations could be turned into a quantitative diagnostic: one could measure how much SVP shifts gate statistics per layer and per task, and use that as a predictor of when selective prompting helps."],"forward_implications":["Frozen Vision Mamba backbones can be competitively adapted with about 1.6M trainable parameters, outperforming full fine-tuning on 7 of 10 HTA datasets in aggregate and matching much larger ViT-B prompting baselines on several datasets.","Prefix-based and image-overlay prompting methods are not the right inductive bias for sequential state-space vision models; token-wise, input-dependent prompts are.","Separating shared cross-layer prompts from per-layer inner prompts captures complementary information, and tuning the balance between them with learned scaling vectors improves accuracy over either path alone.","The largest VTAB-1K gains appear in the Structured group (+13.8 points over VPT), suggesting selective gate activation especially helps tasks that require spatial and relational reasoning."],"supporting_citations":[{"why":"Supplies the Vision Mamba backbone whose selective state-space gates SVP is designed to activate.","marker":"(Zhu et al. 2024)"},{"why":"Defines the Mamba selective state-space layer and the input-dependent B, C, and Delta parameters that make token-wise prompts effective.","marker":"(Gu and Dao 2023)"},{"why":"Provides VPT, the prefix-prompt baseline and deep-prompt design that SVP directly compares against and modifies.","marker":"(Jia et al. 2022)"},{"why":"Provides the DAM-VP baseline and the HTA benchmark protocol used for the main comparisons.","marker":"(Huang et al. 2023)"},{"why":"Provides the SPT baseline whose prompt-initialization strategy is an alternative the paper must beat.","marker":"(Wang et al. 2024)"},{"why":"Provides the AutoVP baseline and automated prompting benchmark used in comparisons.","marker":"(Tsao et al. 2024)"},{"why":"Provides E2VPT, a prompt-tuning baseline in the VPT family that informs the related-work comparison.","marker":"(Han et al. 2023)"},{"why":"Supplies the VTAB-1K benchmark and its natural, specialized, and structured task split.","marker":"(Zhai et al. 2019)"},{"why":"Provides the SA2VP baseline and follows the same benchmark protocol for HTA.","marker":"(Pei et al. 2024)"}],"fun_headline_variants":["Token-wise prompts unlock Vision Mamba's selective gates","Selective prompts let frozen Vision Mamba learn better","SVP: Per-token prompts for efficient Vision Mamba fine-tuning","Input-dependent prompts boost Vision Mamba's discriminative power","Vision Mamba tuned with per-token prompts reaches new SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the accuracy gains come from input-dependent selectivity of the prompters steering Mamba's gates, rather than from the extra per-layer trainable capacity those prompters add; it reports no parameter-matched non-selective baseline.","fun_headline_variants_meta":{"raw":{"variants":["Token-wise prompts unlock Vision Mamba's selective gates","Selective prompts let frozen Vision Mamba learn better","SVP: Per-token prompts for efficient Vision Mamba fine-tuning","Input-dependent prompts boost Vision Mamba's discriminative power","Vision Mamba tuned with per-token prompts reaches new SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1728,"prompt_tokens":1073,"completion_tokens":655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":689,"tokens_out":655,"duration_ms":6830,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:05.506989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a non-selective version of SVP with the same parameter budget and dual-path structure in which each prompt is a learned constant per layer (or per layer group) rather than a function of the input, and compare on HTA and VTAB-1K; if it matches SVP's accuracy, the claim that input-dependent gate activation is the cause is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Vision Mamba backbone whose selective state-space gates SVP is designed to activate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SPT baseline whose prompt-initialization strategy is an alternative the paper must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AutoVP baseline and automated prompting benchmark used in comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides E2VPT, a prompt-tuning baseline in the VPT family that informs the related-work comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SA2VP baseline and follows the same benchmark protocol for HTA."}],"review_version":1}