{"id":"b88d3e95-9bca-43d2-97f7-1803cb67c339","arxiv_id":"2501.03332","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CM3T shows that multi-head vision adapters plus cross-attention adapters can adapt frozen supervised-pretrained video transformers with a fraction of the trainable parameters of full fine-tuning.","lead":"CM3T adds small trainable adapter blocks to frozen video transformers so the backbone stays fixed while new datasets and modalities are learned efficiently. The authors report accuracy close to or above prior state-of-the-art on three interaction datasets while training only 12.8% to 28.9% of the backbone's parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The multi-head vision adapter's gain is not causally tied to spatial chunking: no matched-parameter ablation isolates it, and the fixed-location premise conflicts with egocentric EK-100.","rationale":"Reviewing in good faith: the paper has real experiments, three datasets, two backbones, and ablations of learnable scaling and activation. The central claim is that MHVA enables PETL under supervised pretraining, and the most load-bearing unverified link is the causal role of head-splitting. The paper's own motivation says spatial stability is why head-splitting works, yet the implementation uses window-based splitting in Video Swin (where windows shift), and one of the three datasets, EK-100, is egocentric with a moving camera. This tension makes it plausible that the 39.8% vs 28.7% gap is due to learnable scaling or extra parameters, not spatial chunking. The comparison to AdaptFormer does not control for these factors, and no code or error bars are provided. This is not an objection to the empirical result itself: if the matched-parameter ablation still shows a head-split advantage, the spatial mechanism is supported and CM3T's novelty stands. Until then, the conditional verdict remains appropriate, with the additional condition that the head-split ablation be reported and/or code released.","tokens_in":13121,"tokens_out":8216,"duration_ms":79139,"concrete_test":"Run on EK-100 RGB (and, if possible, a moving-camera dataset such as a Kinetics subset) a matched-parameter ablation: (a) MHVA+PT as reported; (b) a single-head parallel adapter with the same total number of trainable parameters (bottleneck expanded to match the sum of all head-specific down/up projections), the same learnable scaling, ReLU, prefix tuning, and placement; (c) MHVA with head assignments randomized across windows so heads no longer correspond to spatial chunks. If (b) or (c) matches (a), the spatial-chunk mechanism is not the source of MHVA's advantage and the central claim needs revision. If (a) clearly beats both, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty of CM3T is the multi-head vision adapter (MHVA), whose reported benefit is large: with supervised Video Swin-B on EK-100 RGB, MHVA+PT reaches 39.8% vs 28.7% for scaled parallel adapters+PT. The paper explains this by asserting that interaction datasets have almost fixed cameras and that body parts and objects occur in stable spatial locations, so splitting the input along the window/spatial dimension into separate adapter heads lets each head specialize. That explanation is not backed by the implementation or the evidence. In Video Swin, window partitions shift between consecutive blocks, so a given patch is assigned to different window heads across layers; the division is by current windows, not by fixed image coordinates. On EK-100 itself, the camera is egocentric and moves, so objects do not stay in particular spatial locations, yet MHVA still helps. This suggests the observed gain may come from extra adapter capacity or from the learnable scaling factor (their second change), rather than from head-splitting. The only head-split evidence is a comparison to AdaptFormer+PT (28.7%), which differs from MHVA+PT in head-splitting, learnable scaling, and possibly bottleneck/parameter count. No ablation compares MHVA against a single-head adapter with matched total trainable parameters, matched learnable scaling, and matched placement. Without that, the core architectural claim that spatial chunking is what makes supervised PETL work is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CM3T, a parameter-efficient transfer learning framework for video transformers. It adds multi-head vision adapters (MHVA) and cross-attention adapters (CAA) to a frozen backbone, together with an approximation of prefix tuning. The authors evaluate on Epic-Kitchens-100, MPIIGroupInteraction, and UDIVA v0.5, reporting accuracies comparable to full fine-tuning while training only 12.8% (video) or 22.3%/28.9% (multimodal) of the backbone parameters.","tokens_in":13430,"tokens_out":6842,"duration_ms":55068,"significance":"If validated, the central claim would be practically valuable: CM3T would make supervised-pretrained video transformers adaptable with a small number of trainable parameters, without requiring self-supervised pretraining. The paper tests the approach on three datasets with different tasks and two backbones, and includes comparisons to full fine-tuning and to other PETL methods. However, the key architectural novelty (MHVA) is not isolated by a matched-parameter ablation, and the SOTA comparisons are partly confounded. The efficiency numbers are encouraging but the causal explanation of the gain is not yet supported.","major_comments":[{"comment":"The central claim that the multi-head vision adapter (MHVA) is what enables strong performance with supervised pretraining is not established by the evidence. The only direct comparison is between 'MHV A + PT' (39.8%) and 'Adaptformer + PT' (28.7%) in Table 1, but these two configurations differ in three ways: head-splitting, learnable scaling factor, and the total number of adapters/parameters. The paper does not ablate the head-splitting alone: a single-head adapter with the same total bottleneck dimension, the same learnable scaling, and the same placement would isolate the effect of spatial chunking. Without this matched-parameter control, the improvement cannot be attributed to the multi-head design.","section":"§3.2.2, §4.2.1"},{"comment":"The stated mechanism for MHVA is that interaction datasets have almost fixed cameras and stable spatial locations of objects and body parts, so splitting the input along the window dimension allows each head to specialize. This explanation is inconsistent with the implementation: in Video Swin, window partitions shift between consecutive blocks, so a patch is not assigned to a fixed spatial head across layers. Moreover, EK-100 is egocentric and the camera moves, yet MHVA still yields a large gain. The paper should either provide a different rationale or directly test the spatial-specialization hypothesis (e.g., compare fixed versus shifted window assignment, or random head assignment). As written, the offered mechanism is not supported.","section":"§1, §3.2.2"},{"comment":"The SOTA comparison with M&M Mix is confounded by the use of a different backbone architecture and pretraining protocol. The claim of 'comparable and even better results than the state-of-the-art' (abstract) is only true with a CLIP backbone, whereas the 12.8% parameter-efficiency claim refers to Video Swin-B. Please separate these two claims and report a direct comparison where the backbone is held fixed (e.g., full fine-tuning of the same Video Swin-B).","section":"§4.2.1, Table 1"},{"comment":"The percentage of trainable parameters for multimodal input is 22.3% in EK-100 but 28.9% in UDIVA and MPIIGI. The abstract's statement that 'only 22.3% trainable parameters for two additional modalities' is therefore not accurate across the three datasets. Please report all configurations consistently and specify the reference backbone for each percentage.","section":"§4.2.3, Table 4"},{"comment":"The paper does not report several hyperparameters that are necessary to reproduce the experiments: bottleneck dimension r, number of heads in MHVA, initial scaling factor s, gated addition factor λ, dropout values, optimizer, learning rate, and number of training seeds. Without these, the proposed method cannot be evaluated or reproduced from the text. At minimum, the values used in the main experiments should be listed in a table.","section":"§3.2, §4"},{"comment":"No variance or number of seeds is reported for the main results. On small datasets such as MPIIGI, the differences between methods (e.g., 0.901 vs 0.899 mAP) are likely within the noise of a single run. The paper should report mean and standard deviation over multiple runs, or at least clarify that the numbers are single runs.","section":"§4.2.1, Table 1"}],"minor_comments":[{"comment":"There are several typos, including 'UDIV A v0.5' (should be UDIVA) and 'tanH' (should be tanh).","section":"Throughout"},{"comment":"The notation for AdapterFusion is unclear: s' is first defined as a softmax over a row, then used as a scalar multiplier; please clarify dimensions and the concatenation in Eq. (9).","section":"§3.3, Eqs. (7)-(10)"},{"comment":"Figure 2 is dense and the three adapter blocks are hard to distinguish; consider enlarging or adding a schematic of the MHVA head-splitting.","section":"Figure 2"},{"comment":"The sentence 'the cross-attention module proposed by [2] is used in place of the traditional one' is not reflected in the equations. Please specify how the module from [2] is incorporated and how it reduces parameters.","section":"§3.3"},{"comment":"Several symbols in the equations are not defined (e.g., Wq, Wk, Wv, C, λ). Please define all symbols in the text or in a table.","section":"§3.2.1, Eqs. (1)-(5)"}],"recommendation":"major_revision","confidential_remarks":"The SOTA baselines for UDIVA and MPIIGI (FAt transformers) are the authors' own prior work; this is not a reason to reject, but the paper should state this explicitly and ensure that the comparison is fair. Additionally, no code or data is released, which limits reproducibility. The central missing ablation (MHVA vs single-head adapter) should be addressed before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a genuine PETL contribution. The multi-head vision adapter (splitting adapters over spatial windows) and the cross-attention adapter for adding modalities are both new twists on established adapter designs, and the experiments support the main efficiency claim: on three interaction datasets, CM3T gets within a point of full finetuning while training a fifth to a quarter of the parameters. The comparison against AdaptFormer+PT on EK-100 (39.8 vs 28.7) with the same supervised Video Swin-B backbone is a real data point, and the ViViT-B result (38.1 vs 37.4 full finetune) is the kind of small but useful existence proof that adapters can work without self-supervised pretraining.\n\nThe soft spots are real but not fatal. The largest is the causal story. The paper attributes MHVA's gain to head-splitting along stable spatial locations, but the ablation against AdaptFormer changes three things at once: head-splitting, learnable scaling, and bottleneck/parameter counting. A matched-parameter single-head adapter with the same learnable scaling is the missing control. The fixed-camera premise also sits oddly with EK-100 being egocentric; if MHVA helps there, the win may come from capacity or scaling rather than spatial chunking. That weakens the mechanistic claim, not the empirical one.\n\nThe SOTA comparisons are a bit loose. M&M Mix uses a different backbone family, and the FAt transformers baselines on UDIVA and MPIIGI are the authors' own prior work. Neither is disqualifying, and they do provide same-backbone full-finetuning baselines, which is the right control. What is missing is error bars (these numbers are close enough that variance could matter) and code release, which the paper neither mentions nor links.\n\nOverall this is a well-scoped, honest paper. The abstract's 'comparable and even better' is a fair description of the tables. It deserves a serious referee. I'd ask the authors for a matched-parameter ablation of the multi-head split, error bars on the key comparisons, and code. If those are added, I'd be comfortable with it.","headline":"Useful PETL framework with honest empirical claims, but the spatial-chunking explanation for the multi-head adapter is not isolated by the experiments.","tokens_in":13945,"tokens_out":2813,"would_cite":true,"duration_ms":24156,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CM3T claims frozen video transformers can be retuned by plug-in adapters using only 12.8% of backbone parameters for video — 22.3% with two extra modalities — matching or beating full finetuning on three interaction datasets.","keywords":["parameter-efficient transfer learning","multi-head vision adapter","cross-attention adapter","prefix tuning","video classification","multimodal learning","Video Swin Transformer","interaction recognition"],"falsifier":"Permute which spatial windows are assigned to which adapter head and retrain: if the head-wise split still performs as well, the gain is just added parameters and the spatial premise is not doing the work. Alternatively, run the same MHVA-versus-plain-adapter comparison (39.8% versus 28.7% on EK-100) on a dataset with heavy camera or object motion, such as Something-Something, where the fixed-camera assumption breaks; matching performance would refute the claimed mechanism.","tokens_in":12929,"feed_emoji":"🧩","tokens_out":14088,"duration_ms":113399,"temperature":0.7,"pith_summary":"The paper claims that adapter-based transfer learning, which normally only succeeds when the video backbone was pretrained with expensive self-supervised methods such as CLIP or VideoMAE, can be made to work with ordinary supervised checkpoints instead. Its framework, CM3T, freezes a Video Swin Transformer pretrained on Kinetics-400 and attaches two new kinds of trainable plugins: multi-head vision adapters, which give different groups of spatial windows their own adapter heads, and cross-attention adapters, which fold audio, optical flow, or text into the model as additional plugins. With a frozen backbone, CM3T reports 48.2% top-1 accuracy on Epic-Kitchens-100 (full finetuning reaches 48.9%, the three-branch state of the art 49.6%), a state-of-the-art mean MSE of 0.69 on UDIVA, and 0.901 mAP on MPIIGroupInteraction, all while training 12.8% of the backbone's parameters for video alone and 22.3% for two extra modalities. A sympathetic reader would care because it points to a way of reusing the widely available, cheap-to-produce supervised checkpoints instead of forcing users toward self-supervised pretraining.","feed_headline":"12.8% of the parameters matches full finetuning on video tasks","feed_subtitle":"It beats adapter methods that need self-supervised backbones, matching or topping full finetuning on three datasets.","key_machinery":"The load-bearing object is the multi-head vision adapter: a scaled parallel adapter whose input is divided along the spatial window (patch) dimension, with separate down/up projections per head, so that each frame region gets its own fitted transformation, plus a learnable scaling factor $s$ that lets the adapter change activations more forcefully. It is what lets a frozen, supervised-pretrained backbone absorb a new dataset's distribution shift. The second mechanism is the cross-attention adapter, which replaces the adapter's linear layers with cross-attention over a side modality's embedding ($Q = V = \\mathrm{ReLU}(xW_{\\mathrm{down}})$, $K$ = modality embedding), turning a new modality into a self-contained trainable plugin; AdapterFusion then weights the per-modality outputs so the modalities are merged rather than summed. Prefix tuning, kept as the third module, is modified with ReLU activation and gated addition.","core_discovery":"The central claim is that the poor performance of adapters on supervised-pretrained video transformers is not a limit of the adapter idea but of its shape. Existing scaled parallel adapters put the same linear down/up projection on every spatial patch, whereas CM3T splits the input along the window dimension of the Video Swin transformer and gives each spatial chunk its own adapter head with a smaller bottleneck, so different frame regions learn different representations of the dataset shift; swapping in this design lifts the same prefix-tuned recipe from 28.7% to 39.8% on Epic-Kitchens-100. The paper further claims that adding a modality is a pure plug-in operation: a cross-attention adapter whose queries and values come from the vision bottlenecks attends to the new modality's embedding as key, and AdapterFusion merges the per-modality outputs. With three input sources, CM3T gains 8.4 points over its RGB-only version on EK-100, whereas the much larger M&M Mix ensemble gains only 2.9 points over its own RGB-only base. On a second backbone, ViViT-B, the same plugins surpass full finetuning (44.3% versus 37.4%), which the paper reads as evidence that the method is model-agnostic.","pith_inferences":["A control the paper does not run would separate mechanism from parameter count: randomly shuffle which spatial windows each adapter head processes; if accuracy barely changes, the claimed spatial prior is not what carries the gain.","If the fixed-camera premise is the real driver, the head-wise split should transfer to other geometry-stable domains such as fixed-surveillance video or medical imaging, and its edge over plain adapters should shrink on mobile or freely moving-camera footage, a boundary that could be mapped directly.","Because the cross-attention adapter only needs an embedding of the side modality, the same 5.8M-parameter plug-in likely extends to other signals the paper never tries — depth, gaze, skeleton pose, or language instructions — making the framework a generic fusion socket rather than a video-specific trick.","The learnable scaling factor and ReLU-based prefix tuning suggest a training-dynamics story of easier convergence on small data; that predicts the largest advantage in the low-data regime, which could be checked by subsampling the training sets."],"forward_implications":["Supervised checkpoints, not only self-supervised ones, become usable for parameter-efficient transfer: a frozen Video Swin-B pretrained on Kinetics-400 comes within 0.7 points of full finetuning on EK-100 while training 12.8% of the parameters.","Each new modality is a standalone plug-in costing about 5.8M extra trainable parameters, and the added modalities are put to use efficiently: CM3T gains 8.4 points on EK-100 when two modalities join, more than the 2.9-point gain of the much larger M&M Mix ensemble.","Adapter weights transfer across datasets for the same modality, which the paper reports speeds convergence on small datasets: MPIIGI converges in 9 epochs where full finetuning needs 17.","The recipe transfers across architectures: the same plugins on ViViT-B reach 44.3% top-1, surpassing that backbone's full finetuning at 37.4%, supporting the framework's model-agnostic claim."],"supporting_citations":[{"why":"Supplies the unified parallel-adapter-plus-prefix-tuning formulation that CM3T rebuilds into multi-head and cross-attention form.","marker":"[10]"},{"why":"Defines the adapter block (bottleneck down/up projections) whose linear layers MHVA splits along the window dimension and the cross-attention adapter replaces with attention.","marker":"[11]"},{"why":"The AdaptFormer baseline using scaled parallel adapters on a self-supervised backbone; its adapter design at 28.7% on EK-100 is the comparison that motivates MHVA.","marker":"[6]"},{"why":"Video Swin Transformer, the frozen backbone whose window dimension the multi-head adapter divides into per-head processing.","marker":"[22]"},{"why":"M&M Mix, the multimodal state of the art (49.6% on EK-100) whose parameter budget and 2.9-point multimodal gain CM3T is measured against.","marker":"[38]"},{"why":"FAt transformers, the published state of the art on UDIVA and MPIIGI that CM3T matches or beats, and the source of the MMCA module compared against CAA.","marker":"[2]"},{"why":"ST-Adapters, the CLIP-pretraining-dependent PETL baseline that CM3T outperforms under the same supervised protocol.","marker":"[27]"},{"why":"Dual-path adapters, the image-to-video PETL method that works with supervised training but trails CM3T's accuracy.","marker":"[28]"},{"why":"AdapterFusion, the attention-based merger CM3T uses to combine the outputs of per-modality cross-attention adapters.","marker":"[32]"}],"fun_headline_variants":["12.8% of parameters matches full finetuning on video tasks","Add a modality with only 22.3% trainable parameters","Model-agnostic adapters beat state-of-the-art on three video tasks","Plug-in adapters outperform full finetuning without retraining backbone","Splitting spatial adapters lifts epic-kitchens accuracy from 28.7% to 39.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The multi-head vision adapter's advantage rests on interaction videos having roughly fixed cameras so that the same spatial regions reliably hold the same content; without that regularity, or if the pretraining data is too dissimilar to the target, the method has no demonstrated edge over plain adapters.","fun_headline_variants_meta":{"raw":{"variants":["12.8% of parameters matches full finetuning on video tasks","Add a modality with only 22.3% trainable parameters","Model-agnostic adapters beat state-of-the-art on three video tasks","Plug-in adapters outperform full finetuning without retraining backbone","Splitting spatial adapters lifts epic-kitchens accuracy from 28.7% to 39.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3334,"prompt_tokens":1023,"completion_tokens":2311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2206}},"tokens_in":639,"tokens_out":2311,"duration_ms":13991,"temperature":1.0,"reasoning_tokens":2206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:19.394382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute which spatial windows are assigned to which adapter head and retrain: if the head-wise split still performs as well, the gain is just added parameters and the spatial premise is not doing the work. Alternatively, run the same MHVA-versus-plain-adapter comparison (39.8% versus 28.7% on EK-100) on a dataset with heavy camera or object motion, such as Something-Something, where the fixed-camera assumption breaks; matching performance would refute the claimed mechanism.","supporting_citations":[{"cited_title":"Parameter-efficient trans- fer learning for NLP","cited_arxiv_id":null,"evidence_quote":"Defines the adapter block (bottleneck down/up projections) whose linear layers MHVA splits along the window dimension and the cross-attention adapter replaces with attention."},{"cited_title":"Video swin transformer","cited_arxiv_id":null,"evidence_quote":"Video Swin Transformer, the frozen backbone whose window dimension the multi-head adapter divides into per-head processing."},{"cited_title":"Multimodal vision transformers with forced attention for behavior analysis","cited_arxiv_id":null,"evidence_quote":"FAt transformers, the published state of the art on UDIVA and MPIIGI that CM3T matches or beats, and the source of the MMCA module compared against CAA."},{"cited_title":"Dual-path adaptation from image to video transformers","cited_arxiv_id":null,"evidence_quote":"Dual-path adapters, the image-to-video PETL method that works with supervised training but trails CM3T's accuracy."},{"cited_title":"Multilingual Detection of Check-Worthy Claims using World Languages and Adapter Fusion","cited_arxiv_id":"2301.05494","evidence_quote":"AdapterFusion, the attention-based merger CM3T uses to combine the outputs of per-modality cross-attention adapters."}],"review_version":1}