{"id":"385de22c-d8de-4b58-b96f-dee2c64af984","arxiv_id":"2411.11223","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MSTA, a multi-modal spatio-temporal adapter with an LLM-based consistency constraint, achieves strong base-to-novel and few-shot video recognition performance using only a small fraction of trainable parameters.","lead":"This paper introduces MSTA, a parameter-efficient multi-modal adapter that adds a small number of trainable layers to a frozen video-language model, together with a consistency loss based on LLM-generated action descriptions. It reports state-of-the-art results on several video action recognition benchmarks while training only a fraction of the model's parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameters in Table 1 appear selected on the base-to-novel harmonic mean (which includes held-out novel classes); the small MSTA vs AdaptFormer/LoRA gains could be tuning artifacts, so the SOTA claim is not yet established.","rationale":"The reader already flags hyperparameter selection and missing descriptions, and I agree partially, but I weight the Table 6/Table 1 correspondence more heavily: the paper's own ablations choose every final hyperparameter on a metric that includes novel classes, which is direct evaluation-set leakage. The gains over the most relevant same-backbone baselines are small, so this is not a hypothetical risk. The Eq. 14 issue is secondary but real: an un-implementable equation means the proposed consistency loss cannot be independently reconstructed. Both concerns are addressable with released artifacts and a clean tuning protocol. I see no evidence of intent; the issue is about the support for the argument. The conditional verdict remains appropriate rather than outright rejection, because the core architecture is a plausible extension and the reported gap may survive a properly controlled protocol.","tokens_in":15040,"tokens_out":8624,"duration_ms":85474,"concrete_test":"Release the exact DeepSeek prompts/descriptions and code, then re-run the base-to-novel experiment (Table 2) with MSTA hyperparameters chosen on a base-class-only validation split before computing any novel accuracy, giving AdaptFormer and LoRA the same tuning budget; if MSTA's harmonic-mean advantage over these baselines falls within run-to-run noise, the SOTA claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the SOTA claim is that the comparisons in Tables 2-5 are fair. The most insecure part is hyperparameter selection: in Section 4.3 (Table 6), the paper tunes adapter dimension, lambda, alpha, N, and layer placement on the base-to-novel harmonic mean, and the final settings in Table 1 (Dims=256, lambda=0.005, alpha=1.0, N=2, Layers=1-12) match the values that maximize HM in Table 6. Because the novel split is held out, optimizing HM leaks test information into model selection. This matters because the same-backbone advantage over AdaptFormer/LoRA is modest (HM +0.6 to +1.7 in Table 2), so the reported 'outstanding' results could be a tuning artifact. Eq. 14 compounds the problem: as printed it cannot be the loss actually minimized, since w_c and D_c are both attributed to the pre-trained branch and the second term subtracts a vector rather than a cosine similarity; the real regularization term is not reconstructible from the paper. Both issues are addressable, but until they are, the central numerical claim lacks independent support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MSTA, a parameter-efficient multi-modal spatio-temporal adapter for transferring video-language foundation models (ViCLIP) to video action recognition. The adapter is inserted into the higher layers of both the text and video encoders, with a shared projection layer for cross-modal alignment and separate spatial/temporal up-projections in the video branch. To preserve generalization, the authors add a spatio-temporal description-guided consistency constraint that enforces agreement between the trainable text branch and a frozen pre-trained text branch fed with LLM-generated action descriptions. The method is evaluated on zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised recognition across several datasets, reporting state-of-the-art results while training only 2-7% of the model parameters. The paper includes extensive ablations on adapter design, hyperparameters, and description counts.","tokens_in":15296,"tokens_out":5970,"duration_ms":52155,"significance":"If the claims are substantiated, this is a useful contribution to efficient transfer learning for video-language models: it demonstrates that a lightweight adapter with a regularization constraint can match or exceed full fine-tuning and existing PEFT baselines on multiple benchmarks. The experimental coverage is broad (six datasets, four evaluation protocols), and the comparisons against AdaptFormer and LoRA on the same ViCLIP backbone are appropriate. The paper also makes an honest observation in Table 6e that LLM hallucination degrades performance as the number of descriptions grows, which tempers the method's dependence on description quality. However, the significance of the claimed state-of-the-art results is currently undercut by (i) an incorrectly specified consistency loss in Eq. (14), (ii) hyperparameter selection using the held-out novel-class harmonic mean, and (iii) the absence of released descriptions and error bars in most tables. These issues are fixable, but until addressed the numerical claims lack independent support.","major_comments":[{"comment":"The consistency loss as printed in Eq. (14) is not a valid loss function. The first term has the scalar dot product w_c · D_s in the numerator but a denominator containing ||w_c · D_t||, which is the norm of a vector (or a scalar, depending on interpretation), and the second term w_c / (||w_c|| ||D_t||) is a vector, not a scalar loss. Dimensionally the two terms cannot be combined into a scalar loss. Moreover, the text states that the constraint enforces consistency between the trainable branch and the pre-trained branch, but Eq. (14) only involves quantities attributed to the pre-trained branch (w_c, D_s^c, D_t^c). The actual regularization term therefore cannot be reconstructed from the paper. Please provide a dimensionally consistent, unambiguous definition that specifies which embeddings come from the trainable branch and which from the pre-trained branch, and how the cosine distance is computed. This is load-bearing because the consistency constraint is one of the two central contributions and its benefits are reported in Tables 2-5.","section":"3.3, Eq. (14)"},{"comment":"The hyperparameters in Table 1 appear to be selected using the base-to-novel harmonic mean reported in Table 6. Since the harmonic mean includes accuracy on novel classes, which are held out during training, this selection procedure leaks test information into model choice. The final settings (lambda=0.005, alpha=1.0, N=2, Dims=256, Layers=1-12) coincide with the values that maximize HM in Table 6. Given that the improvements over AdaptFormer and LoRA in Table 2 are modest (e.g., HM gains of +0.6 to +1.7), the reported ranking may partly be a tuning artifact. The authors should either tune hyperparameters on a validation split that excludes novel classes, or report the sensitivity of the main comparisons to reasonable hyperparameter variations to show that the conclusions are robust.","section":"4.3, Table 6, Table 1"},{"comment":"The spatio-temporal descriptions generated by the LLM are a load-bearing input to the consistency constraint, but the full set of descriptions is not released in the paper or supplementary material. Table 6e shows that performance degrades when N increases from 2 to 8, which the authors attribute to LLM hallucination; this indicates that the benefit of the consistency loss depends heavily on the exact descriptions and the generation procedure. Without releasing the descriptions and specifying the prompt details, LLM version, and decoding settings, the consistency-loss contribution is not reproducible and its claimed effect cannot be independently evaluated.","section":"3.3, Table 6e"},{"comment":"Tables 2, 3, and 5 report no standard deviations or multiple-seed runs, while several differences between MSTA+LCC and the strongest PEFT baselines are small (e.g., Table 2 HM on UCF-101: 82.9 vs. 82.3; Table 3 HMDB-51 K=16: 70.4 vs. 70.1). Without error bars or significance tests, it is unclear whether these differences are meaningful. The variance reporting in Table 4 (mean ± std) is a good model and should be extended to the other main tables.","section":"Tables 2, 3, 5"}],"minor_comments":[{"comment":"The introduction lists six benchmark datasets including ActivityNet, but Section 4.1 lists only five datasets and no ActivityNet results are reported anywhere; please reconcile this inconsistency.","section":"Section 1 vs. 4.1"},{"comment":"The paper refers to the consistency constraint as a 'cosine distance' but Eq. (14) is not the standard cosine distance; please align the terminology with the corrected equation.","section":"Section 3.3 text after Eq. (14)"},{"comment":"The sentence 'we improve the recognition accuracy of the novel category on SSv2 from 11.5 to 15.8 (an increase of 30 37%)' contains a garbled fragment '30 37%'; the increase is 37% and '30' should be removed.","section":"Section 4.2, SSv2 comparison"},{"comment":"The reference to 'Table 4a' should be 'Table 6a', since the component-wise ablation is shown in Table 6.","section":"Section 4.3, first paragraph"},{"comment":"The softmax expression in Eq. (15) has an unmatched parenthesis and is missing a closing brace for the logarithm; please correct the typesetting.","section":"Eq. (15)"},{"comment":"There are spelling errors in the author block: 'Nanjing Univerisy' should be 'Nanjing University' and 'avaliable' should be 'available'. Also, 'A5' in Tables 2 and 3 is not defined; if it refers to AdaptFormer, please use the full name.","section":"Author affiliations and footnote"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to acceptance is not the core idea but the evidence supporting the SOTA claim. The hyperparameter selection on held-out novel classes and the invalid Eq. (14) are fixable with a corrected formulation and a validation-based tuning protocol. The paper would also be much stronger if the LLM-generated descriptions were released. This is a publishable contribution after major revision, provided the authors address these reproducibility and evaluation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, incremental PEFT paper for ViCLIP. The shared adapter idea and the description-guided consistency loss are reasonable, and the experimental coverage is thorough. But the main quantitative claim is not yet trustworthy as stated: the consistency loss equation is ambiguous to the point of being wrong, and the hyperparameters appear to be selected on the base-to-novel harmonic mean, which includes the held-out novel classes.\n\nWhat's actually new: the specific combination of a shared projection adapter between text and video branches, with separate spatial and temporal up-projections, and an LLM-generated description consistency loss. That combination is not in the cited literature. The paper evaluates on four tasks and six datasets, with consistent improvements over AdaptFormer and LoRA on the same ViCLIP backbone. The ablations are unusually complete, covering adapter variants, dimensions, scaling, weighting, description count, and layer placement. The parameter count is indeed small (about 8.7M trainable, roughly 7% of the model), and the fully-supervised K400 result (82.2) is competitive with methods using many more trainable parameters.\n\nSoft spots. First, Eq. (14) is not a real loss as printed. The text says the constraint ties the learnable branch to the pre-trained branch, but the equation uses only pre-trained branch quantities: w_c and both D_s and D_t are all attributed to the pre-trained encoder. The second term lacks a cosine distance structure and the denominator in the first term mixes D_t with D_s. The actual loss is not reconstructible from the paper. That is a genuine reproducibility problem. Second, the hyperparameter selection leaks test information. The ablations in Table 6 tune lambda, alpha, N, dimension, and layer range on the base-to-novel HM, which averages base and novel accuracy. Choosing the settings that maximize HM on the same split you report as the result means the novel numbers are partly fitted. Given that the gains over AdaptFormer/LoRA are modest (HM +0.6 to +2.0), a skeptic could explain the difference by tuning alone. Third, the LLM-generated descriptions are not released, and the paper itself notes that N=2 is a compromise because larger N triggers hallucination. That makes the consistency loss dependent on an unreleased input.\n\nNone of these are fatal, but they shift the burden back to the authors. The architecture and idea are credible, and the parameter-efficiency claim is plausible. What is not yet established is the specific SOTA improvement.\n\nThis paper is worth a serious referee. It should go to peer review with a request for the corrected loss, a validation protocol that tunes on base classes or a separate split, and release of code and descriptions. The audience is researchers working on efficient video-language adaptation; for that audience the paper is useful but needs revision.","headline":"Solid incremental PEFT paper for ViCLIP, but the SOTA claim needs a corrected loss equation and a validation protocol that doesn't tune on held-out classes.","tokens_in":15817,"tokens_out":4147,"would_cite":false,"duration_ms":37599,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-modal spatio-temporal adapter that trains only 2-7% of the parameters matches or beats full fine-tuning on video action recognition while keeping generalization to unseen classes.","keywords":["parameter-efficient transfer learning","video action recognition","video-language foundation model","multi-modal adapter","spatio-temporal description","consistency constraint","base-to-novel generalization","few-shot learning"],"falsifier":"A decisive test is to rerun the base-to-novel protocol on UCF-101 and Something-Something v2 with the number of LLM descriptions set to N=8 and compare with the paper's N=2 result: the paper's own hallucination explanation predicts the harmonic mean should drop, so if it rises or stays flat the claimed mechanism is not the cause of the gains. Equally, replacing the LLM descriptions with random sentence-length noise of the same template should collapse performance if the semantic content is load-bearing.","tokens_in":14865,"feed_emoji":"🎬","tokens_out":7621,"duration_ms":68287,"temperature":0.7,"pith_summary":"Pre-trained video-language models can be adapted to action recognition without unfreezing their weights. The paper proposes the Multi-modal Spatio-Temporal Adapter (MSTA), a set of lightweight projection blocks inserted into the top layers of both the text and video encoders, plus a shared projection layer that aligns the two modalities. To keep the model from overfitting to the small training set, a spatio-temporal description-guided consistency constraint forces the trainable text branch to stay close to the frozen branch when both are fed LLM-generated descriptions of the action's visual appearance and temporal steps. The authors report that this setup beats full fine-tuning and prior efficient-transfer methods on zero-shot, few-shot, base-to-novel, and fully-supervised video action recognition while training only 2-7% of the original parameters. If the claim holds, parameter-efficient adapters are not just a compute-saving compromise but a way to improve generalization to unseen classes.","feed_headline":"Adapter beats full fine-tuning with 2-7% of the parameters","feed_subtitle":"Video-language adapter tops zero-shot, few-shot, and base-to-novel action recognition with far fewer trainable weights.","key_machinery":"The load-bearing machinery is the shared projection layer inside MSTA. Instead of independent uni-modal adapters, MSTA routes both text and video features through a common learned projection, so the shared space receives gradients from both modalities and explicitly aligns text and video representations. Around it, each video adapter has a down-projection, a shared middle projection, and two up-projections (spatial linear, temporal 3D convolution); each text adapter has down, shared, and up projections. The second load-bearing mechanism is the consistency loss, which anchors the trainable text branch to the frozen one by requiring template embeddings to stay close to embeddings of LLM-generated descriptions of the action's visual appearance and temporal steps. This acts as a regularizer against overfitting and catastrophic forgetting.","core_discovery":"The paper's central claim is that adapting a frozen video-language model with MSTA plus a spatio-temporal description-guided consistency constraint yields better accuracy than full fine-tuning across zero-shot, few-shot, base-to-novel, and fully-supervised video action recognition, with only 2-7% of the parameters trainable. MSTA inserts lightweight down/up projection blocks into the upper layers of both the video and text encoders, shares one projection layer between the two branches so gradients from both modalities pull the representations into a common space, and splits the video up-projection into spatial (linear) and temporal (3D convolution) paths. The consistency constraint feeds class templates to the trainable text branch and LLM-generated spatial and temporal descriptions to the frozen text branch, then penalizes the cosine distance between their embeddings; the total loss is $\\mathcal{L} = \\mathcal{L}_{\\mathrm{CE}} + \\alpha \\mathcal{L}_{\\mathrm{CC}}$. On Kinetics-400, HMDB-51, UCF-101, Something-Something v2, and Kinetics-600, the paper reports the best harmonic mean between base and novel classes and the best few-shot and zero-shot accuracies among the compared methods, at 8.7M trainable parameters.","pith_inferences":["Beyond the paper: the N=2 optimum in the authors' ablation implies description quality matters more than quantity; an automatic description-filtering or ensembling step that discards hallucinated sentences could push base-to-novel performance higher than the reported numbers.","Beyond the paper: because the consistency loss only needs paired template and description embeddings, it could be applied with description sources other than LLMs, such as video subtitles or automatically mined caption sets, removing the dependency on a specific external language model.","Beyond the paper: a shared projection layer that receives gradients from both modalities suggests the adapter could be reused for cross-modal retrieval or open-vocabulary video classification with the same trained weights, since text-video alignment is the explicit optimization target."],"forward_implications":["If MSTA holds, full fine-tuning of video-language models becomes unnecessary for action recognition: adapter-only training with 2-7% of parameters gives equal or better accuracy and better generalization to unseen classes.","The consistency constraint gives a practical recipe for injecting external knowledge from LLM descriptions into frozen multi-modal encoders without unfreezing them, which should transfer to other video tasks such as temporal action localization or video retrieval.","Because the shared projection gets gradients from both modalities, the same adapter design can be expected to improve text-video alignment in any contrastively trained video-language model, not only the ViCLIP backbone used here.","In few-shot regimes the paper reports gains of several points on Something-Something v2 (for example, 17.5 versus 15.4 at K=16), suggesting that even very small labeled video sets can be exploited without catastrophic forgetting.","Fully-supervised Kinetics-400 results improve from 79.9 top-1 with the frozen backbone to 82.2 with MSTA plus the consistency constraint, showing the adapter also helps when training data is plentiful."],"supporting_citations":[{"why":"provides the frozen video-language backbone (ViCLIP) that the adapters are inserted into.","marker":"[36]"},{"why":"supplies the standard down-projection/up-projection adapter design that MSTA extends with shared and spatio-temporal branches.","marker":"[16]"},{"why":"is a parameter-efficient fine-tuning baseline the paper compares against in all four evaluation settings.","marker":"[17]"},{"why":"is an adapter-based vision transformer baseline used for comparison in video transfer.","marker":"[5]"},{"why":"introduces LLM-generated spatio-temporal descriptors for video recognition, the closest prior idea and a comparison method.","marker":"[6]"},{"why":"establishes the temporal-modeling baseline with cross-frame attention that MSTA competes with.","marker":"[24]"},{"why":"motivates the consistency constraint as a way to preserve generalization during prompt or adapter learning.","marker":"[30]"},{"why":"demonstrates a multi-modal adapter with a shared feature space, grounding the shared projection mechanism.","marker":"[40]"},{"why":"is the large language model queried to generate the spatial and temporal description sentences used by the consistency loss.","marker":"[7]"}],"fun_headline_variants":["Adapter with 2-7% trainable params beats full fine-tuning","MSTA: 2-7% params, top zero-shot and few-shot video recognition","Tiny adapter trumps full fine-tuning on video-language tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the generated descriptions are the right semantic anchors; if they misdescribe or fail to distinguish classes, the consistency constraint will pull the trainable branch toward wrong text embeddings and reduce accuracy, which is exactly the degradation the authors observe when they increase the number of descriptions from 2 to 8.","fun_headline_variants_meta":{"raw":{"variants":["Adapter with 2-7% trainable params beats full fine-tuning","MSTA: 2-7% params, top zero-shot and few-shot video recognition","Tiny adapter trumps full fine-tuning on video-language tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2976,"prompt_tokens":1058,"completion_tokens":1918,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":1854}},"tokens_in":674,"tokens_out":1918,"duration_ms":13573,"temperature":1.0,"reasoning_tokens":1854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:46:45.102494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to rerun the base-to-novel protocol on UCF-101 and Something-Something v2 with the number of LLM descriptions set to N=8 and compare with the paper's N=2 result: the paper's own hallucination explanation predicts the harmonic mean should drop, so if it rises or stays flat the claimed mechanism is not the cause of the gains. Equally, replacing the LLM descriptions with random sentence-length noise of the same template should collapse performance if the semantic content is load-bearing.","supporting_citations":[{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"is a parameter-efficient fine-tuning baseline the paper compares against in all four evaluation settings."},{"cited_title":"Internvid: A large-scale video-text dataset for multimodal understanding and generation","cited_arxiv_id":null,"evidence_quote":"provides the frozen video-language backbone (ViCLIP) that the adapters are inserted into."},{"cited_title":"Parameter-efficient transfer learning for NLP","cited_arxiv_id":null,"evidence_quote":"supplies the standard down-projection/up-projection adapter design that MSTA extends with shared and spatio-temporal branches."},{"cited_title":"Adaptformer: Adapt- ing vision transformers for scalable visual recognition","cited_arxiv_id":null,"evidence_quote":"is an adapter-based vision transformer baseline used for comparison in video transfer."},{"cited_title":"OST: refining text knowledge with optimal spatio-temporal descriptor for general video recognition","cited_arxiv_id":null,"evidence_quote":"introduces LLM-generated spatio-temporal descriptors for video recognition, the closest prior idea and a comparison method."},{"cited_title":"Expanding language-image pretrained models for general video recognition","cited_arxiv_id":null,"evidence_quote":"establishes the temporal-modeling baseline with cross-frame attention that MSTA competes with."},{"cited_title":"Consistency-guided prompt learning for vision-language models","cited_arxiv_id":null,"evidence_quote":"motivates the consistency constraint as a way to preserve generalization during prompt or adapter learning."},{"cited_title":"MMA: multi-modal adapter for vision-language models","cited_arxiv_id":null,"evidence_quote":"demonstrates a multi-modal adapter with a shared feature space, grounding the shared projection mechanism."}],"review_version":1}