{"id":"3b5cf3ec-29d4-4e1c-8875-5594eb8babcd","arxiv_id":"2412.11509","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Skip Tuning adapts CLIP by fine-tuning only the deep layers on a sampled subset of class tokens per image, beating prompt tuning and adapters in accuracy and efficiency.","lead":"This paper proposes Skip Tuning, a way to adapt CLIP image-text models to new tasks by fine-tuning only the deep layers while caching features from frozen shallow layers and training each image against a random subset of class labels. The method is reported to beat prompt tuning and adapter methods on 11 benchmarks while using far less time and memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameters ω=6, r=0.5, λ=0.3 are tuned directly on the same 11-dataset benchmark used for final results, so the reported accuracy advantage is in-sample and not yet a fair estimate for unseen tasks.","rationale":"The reader's weakest_assumption focuses on the FS/GD metrics being measured on only two datasets and then assumed to extend to all 11. That is a real concern, but I see an even more decisive issue: the three method hyperparameters are selected by optimizing the exact benchmark metric that is later used to claim superiority. This makes the reported accuracy comparison in-sample for Skip Tuning, while the baselines use hyperparameters fixed by their original papers. If the hyperparameters were chosen on a held-out validation split, the central effectiveness claim would be much stronger. The efficiency claim is less affected because the memory/time reductions are structurally guaranteed by skipping layers and class tokens; a concrete test of the effectiveness claim is to re-run the full protocol with validation-based hyperparameter selection. I therefore agree partially with the reader: the FS/GD generalizability issue is related, but the absence of a held-out validation step is the load-bearing flaw. The reader's CONDITIONAL verdict remains appropriate; if the proposed test shows the gains persist, the verdict could move to ACCEPT, but until then the central claim is not fully established. I am not arguing the method is wrong or that the authors acted improperly; the paper is clearly written and the code is released, both of which are positive. The concern is purely about whether the reported accuracy advantage is an honest estimate of performance on unseen tasks.","tokens_in":18073,"tokens_out":5100,"duration_ms":54418,"concrete_test":"Run a held-out hyperparameter selection protocol. Concretely: (1) split each dataset's base classes into a training split and a validation split (or reserve e.g. ImageNet, DTD, and EuroSAT purely for validation); (2) tune ω, r, and λ on the validation split only; (3) freeze the selected hyperparameters and evaluate on the untouched base/new and cross-dataset splits exactly as in Tables 1 and 6; (4) report the mean and standard deviation over at least 5 seeds. If the validation-selected hyperparameters differ from ω=6, r=0.5, λ=0.3, or if the resulting average H advantage over PromptSRC shrinks below the current 1.14-point gap, the in-sample tuning concern is confirmed and the central effectiveness claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Skip Tuning is both more effective and more efficient than prompt tuning and adapter baselines. The efficiency half is structurally plausible: LSkip and CSkip genuinely shorten the per-image forward/backward path. The effectiveness half, however, rests on the choice of three hyperparameters (ω, r, λ), and the paper selects them by maximizing the average harmonic-mean accuracy on the very 11 base/new splits that are later reported as the headline results. Specifically, Section 3.3 and Figure 4 sweep ω over {2,4,6,8,10}, r over {0.1,...,1.0}, and λ over {0.01,...,1.0}, and pick ω=6, r=0.5, λ=0.3 based on average H across all 11 datasets. This is an in-sample model-selection procedure: the same test splits are used both to choose the hyperparameters and to report the final 81.11 H versus 79.97 for PromptSRC. It is therefore possible that part or all of the claimed accuracy edge reflects tuning on the evaluation benchmarks rather than a general property of skipping layers and class tokens. The motivation for skipping is also supported by FS/GD measurements on only two datasets (Caltech101, Food101) in Figure 2(b,c), so there is no independent evidence that ω=6 transfers to the other nine tasks. Since the method has only three tunable numbers, a held-out validation protocol is easy to implement and should be reported before the effectiveness claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Skip Tuning, an adaptation method for CLIP that reduces feature-gradient propagation flows (FGPFs) through two operations: Layer-wise Skipping (LSkip), which caches features at the omega-th layer and fine-tunes only the remaining deep layers, and Class-wise Skipping (CSkip), which samples a subset of class tokens per training image when computing the image-text matching loss. The authors argue that freezing shallow layers and sampling class tokens yields both higher accuracy and much higher efficiency than prompt tuning and adapter-based baselines. They support this with: (i) FS/GD metrics showing shallow layers and most class tokens contribute little during fine-tuning on two datasets; (ii) experiments on 11 benchmarks across base-to-new, cross-dataset, domain generalization, and few-shot settings, reporting higher harmonic-mean accuracy with substantially lower time and memory than prior methods; (iii) ablations showing that LSkip and CSkip both improve on the full fine-tuning baseline.","tokens_in":18424,"tokens_out":3616,"duration_ms":33070,"significance":"If the reported results hold, Skip Tuning is an conceptually simple and parameter-free (beyond three hyperparameters) alternative to prompt tuning and adapter modules, with structurally plausible efficiency gains: skipping layers and class tokens directly shortens the per-example forward/backward path. The paper ships a public code repository, and the efficiency advantages are reported consistently across multiple evaluation settings. However, the effectiveness claim is currently weakened by the in-sample selection of the three hyperparameters on the very benchmarks used for the final results, and by the absence of any variance or significance measures despite the paper stating that all results are averaged over three runs. The method's novel contribution, LSkip and CSkip, is a genuine and potentially useful idea, but the evidence as presented does not yet establish that the accuracy edge over prior art generalizes beyond the tuned configuration.","major_comments":[{"comment":"The hyperparameters ω=6, r=0.5, λ=0.3 are selected by maximizing the average harmonic-mean accuracy on the same 11 datasets that are then reported as the final results in Table 1. This is an in-sample model-selection procedure: the test splits are used both to choose the hyperparameters and to report the headline 81.11 H versus 79.97 for PromptSRC. Part or all of the claimed accuracy advantage may reflect tuning on the evaluation benchmarks rather than a general property of skipping layers and class tokens. The authors should either choose hyperparameters on a separate validation split (e.g., a subset of datasets or a train/validation split of each dataset) or report performance across the full hyperparameter grid to demonstrate that the result is not an artifact of selection. Given that the method has only three tunable scalars, this is straightforward to address.","section":"§3.3, Figure 4"},{"comment":"The motivation for skipping layers and class tokens rests on FS and GD metrics computed on only two datasets (Caltech101 and Food101). The paper then applies the fixed choice ω=6 across all 11 evaluation datasets. No evidence is provided that the sensitivity profile measured on these two datasets transfers to the other nine tasks, which include fine-grained (Aircraft, Cars) and scene/dataset categories with very different properties. The Euclidean-distance heuristics in Eqs. (3) and (4) are also not validated against any downstream performance criterion. The authors should either compute FS/GD on more datasets or, better, connect the FS/GD ranking to held-out validation performance before the skipping strategy is fixed.","section":"§2.2, Figure 2(b,c)"},{"comment":"The paper states that 'all experimental results are the average of 3 runs with different seeds,' but Tables 1–3 and 5 report only point estimates without standard deviations or error bars. Consequently, the reader cannot assess whether the reported improvements (e.g., 1.14% H over PromptSRC in Table 1, or 0.02% for CSkip-only in Table 4) are statistically meaningful. The authors should report standard deviations or confidence intervals for at least the main base-to-new results and the ablations.","section":"§3.1 (Implementation details)"},{"comment":"The text states that 'both LSkip and CSkip contribute to performance improvement,' but the ablation table shows that CSkip alone (Base 84.77, New 74.48, H 79.29) yields essentially no harmonic-mean improvement over the FT baseline (H 79.27), while LSkip alone yields H 81.00. The only substantial benefit of CSkip alone is efficiency (time drops from 1002s to 440s, memory from 1846M to 919M). The claim should be corrected to state that LSkip is the main accuracy driver and that CSkip contributes primarily to efficiency, with only a marginal accuracy effect when used alone.","section":"§3.3, Table 4"},{"comment":"CSkip samples a subset of class tokens for each image, but the paper does not specify whether the softmax normalization in Eq. (2) is over the sampled subset (with M replaced by the per-image m) or over all M classes using the cached text features. This distinction is essential for reproducibility: if the denominator still sums over all classes, the loss is unchanged except for the absence of gradients for unsampled classes; if it sums over the sampled subset, the loss itself changes. The implementation in the released code may resolve this, but the manuscript should state the choice explicitly.","section":"§2.3, Eq. (2) and Eq. (7)"}],"minor_comments":[{"comment":"The text refers to 'Table 6' for cross-dataset and domain generalization results, but in the main paper these are Tables 2 and 3; the supplementary also contains Tables 6 and 7. The cross-references are inconsistent and should be harmonized.","section":"§3.2"},{"comment":"The text refers to 'Table 8' for adapter-based comparisons, but the corresponding table in the main text is Table 5. Please align the numbering.","section":"§3.4"},{"comment":"There are typos: 'training apochs' should be 'training epochs', and in Supplementary B 'target ddomains' should be 'target domains'.","section":"Supplementary A"},{"comment":"The caption states that '× indicates the performance improvement over the state-of-the-art,' but the figure itself is not shown in the manuscript text and the exact meaning of the multiplicative factors is not defined. Please clarify.","section":"§1, Figure 1 caption"},{"comment":"The column header 'ProSRC.' should be 'PromptSRC' for consistency with the text and other tables; the same abbreviation appears in Tables 2 and 3.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the in-sample selection of ω, r, and λ on the evaluation benchmarks. This is a load-bearing issue: the paper's effectiveness claim is a comparison of accuracy, and the accuracy margin may be entirely due to tuning on the test splits. The paper's own ablation already suggests LSkip is the main accuracy driver, so a held-out validation protocol would likely preserve the efficiency story but could reduce the accuracy margin. I would encourage the editor to require the authors to report a validation-based hyperparameter selection or a robustness analysis across the hyperparameter grid before acceptance. The manuscript also has several internal numbering inconsistencies (Table 2 vs 6, Table 5 vs 8) that suggest a rushed revision; these are minor but should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main news is that you can fine-tune only the deep layers of CLIP, cache the shallow features once, and sample a subset of class tokens per image during training, and this beats prompt tuning and adapter baselines in both accuracy and cost. The efficiency story is real: LSkip and CSkip cut the forward/backward path dramatically, and the reported time/memory numbers line up with that. The accuracy story is mostly credible but has a wrinkle: the three hyperparameters (omega, r, lambda) are chosen by sweeping on the same 11 base/new splits used for the headline 81.11 H, so part of the edge over PromptSRC likely comes from in-sample selection.\n\nThe method itself is simple and well explained. The combination is new: feature caching appears in Tip-Adapter, partial fine-tuning is old, but putting them on the FT baseline with the image-conditioned class sampling is not in the cited literature. The ablations in Table 4 show both components help, and the gain is consistent across 11 datasets. The paper also does a useful service by showing that careful full fine-tuning beats CoOp, which many prompt-tuning papers implicitly assume is impossible. Code is released; that counts for something.\n\nThe soft spots are real but not fatal. Section 3.3 sweeps omega, r, and lambda on the same 11 datasets used for the final tables. To the authors' credit, the sensitivity curves for r and lambda are fairly flat, so the method is probably not brittle, but omega=6 was picked by maximizing average H on those test splits. There is no held-out validation and no error bars despite averaging three runs. The motivation metrics FS/GD come from only two datasets and are assumed to transfer to the other nine. The CSkip sampling procedure is also underspecified: are the remaining classes sampled independently without replacement, and how is the softmax over the sampled subset normalized? Finally, the efficiency comparison is not entirely apples-to-apples, since the FT baseline uses 20 epochs while some prompt baselines use 10.\n\nNone of these are load-bearing flaws. The efficiency advantage is structural, and the accuracy advance is plausible even if the exact 1.14-point H gap may shrink under a fair protocol. The paper deserves a serious referee, and I would accept it with a request for held-out hyperparameter selection, error bars, and a precise CSkip description. For anyone working on VLM adaptation, this is worth citing and worth reading.","headline":"Skip Tuning is a genuinely simpler and faster way to adapt CLIP, and the accuracy gains are plausible but partly in-sample because the three hyperparameters are tuned on the same 11 benchmarks used for the headline results.","tokens_in":18910,"tokens_out":2870,"would_cite":true,"duration_ms":27502,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reducing the length and width of feature-gradient propagation flows is the key to efficient vision-language adaptation.","keywords":["vision-language models","CLIP","prompt tuning","parameter-efficient fine-tuning","layer-wise skipping","class-wise skipping","base-to-new generalization","few-shot learning"],"falsifier":"Measure Feature Sensitivity on all 11 datasets before fixing $\\omega$; if any dataset shows large FS values in the first six layers, the LSkip premise fails there. A direct test is to run Skip Tuning on a dataset where shallow layers carry task-specific structure, such as a texture- or edge-dominated benchmark, and compare base and new accuracy against full fine-tuning; a large accuracy drop would falsify the claim that shallow layers are universally dispensable.","tokens_in":17892,"feed_emoji":"⚡","tokens_out":9241,"duration_ms":70901,"temperature":0.7,"pith_summary":"This paper argues that the standard recipe for adapting vision-language models—freezing the model and learning a few prompt vectors—sacrifices accuracy without actually saving much memory or time. The authors claim the real lever is the feature-gradient propagation flow of full fine-tuning: most shallow layers and most class tokens contribute little per training image, so both can be skipped. Their Skip Tuning method caches features from the first $\\omega$ layers, fine-tunes only the deep layers, and samples a small subset of class tokens per image. On 11 benchmarks they report higher accuracy than prompt tuning and adapter methods while using far less time and memory. The claim, if right, would make the pre-trained model itself the adapter, with no extra parameters.","feed_headline":"Skip Tuning beats prompt tuning at a fraction of the cost","feed_subtitle":"Freezing shallow layers and sampling class tokens per image lifts accuracy while cutting memory and training time","key_machinery":"The central object is the Feature-Gradient Propagation Flow (FGPF): the forward flows of image and text features and the backward flows of gradients through the vision and text encoders. The paper models total cost as $C_{\\mathrm{total}} = N \\times (C_V + C_T \\times M)$, where $N$ is the number of layers, $M$ the number of class tokens, and $C_V, C_T$ the per-layer costs of the vision and text encoders, so layer count is the FGPF length and class-token count the FGPF width. Two diagnostic metrics justify the skipping: Feature Sensitivity (FS), the Euclidean distance between a layer's feature before and after fine-tuning, identifies shallow layers as dispensable, and Gradient Dependence (GD), the change in feature gradients when a class token is removed, identifies most class tokens as dispensable for any given image. LSkip and CSkip operationalize these observations by cutting length and width respectively.","core_discovery":"The paper's central claim is that reducing both the length and the width of the feature-gradient propagation flows (FGPFs) of the full fine-tuning baseline is the key to effective and efficient knowledge transfer. Length means the number of transformer layers that features and gradients traverse; width means the number of class tokens used in the image-text matching loss for each training image. Layer-wise Skipping (LSkip) caches intermediate features after the first $\\omega$ layers and fine-tunes only the remaining deep layers; Class-wise Skipping (CSkip) samples a small image-conditioned subset of class tokens, keeping the top similar classes and probabilistically including others. The authors report that this two-way skipping improves base-to-new, cross-dataset, and domain generalization accuracy over state-of-the-art prompt tuning while cutting time and memory by large factors, and that it also beats adapter-based methods such as LoRA and CLIP-adapter.","pith_inferences":["The FS and GD diagnostics could be reused as a cheap dataset profiler: measuring layer and class-token importance on a small validation set before choosing $\\omega$, $r$, and $\\lambda$ could let the skipping hyper-parameters be selected per task rather than fixed.","The same length-and-width reduction logic may transfer to other large frozen models beyond CLIP, wherever shallow layers and many label tokens are redundant for a target task.","If shallow-layer features are genuinely task-agnostic, caching them once per dataset would let many tasks share the same cache, making per-task fine-tuning nearly as cheap as linear probing.","The reported insensitivity of accuracy to $r$ between 0.4 and 1.0 suggests the benefit of CSkip is mainly distraction removal rather than hard class selection; a direct test is whether random class dropping without similarity sorting gives the same gain."],"forward_implications":["Fine-tuning only the last $N-\\omega$ layers of CLIP, with features cached from frozen shallow layers, preserves or improves accuracy relative to full fine-tuning on the tested benchmarks.","Sampling a small image-conditioned subset of class tokens per training image (with $r=0.5$ and $\\lambda=0.3$) cuts memory and time while improving generalization, because it filters redundant text features.","Skip Tuning outperforms prompt-tuning methods on base-to-new generalization (known and unseen classes), cross-dataset transfer, and domain generalization, and outperforms LoRA and CLIP-adapter in both accuracy and efficiency.","Efficiency gains grow with dataset size: on cross-dataset ImageNet training, Skip Tuning reports roughly 44 times less time and 21.5 times less memory than PromptSRC.","No learnable prompt vectors or adapter modules are introduced, so the adapted model is still the pre-trained weights plus cached intermediate features."],"supporting_citations":[{"why":"Supplies the pre-trained CLIP model and the zero-shot image-text matching setup used as the testbed.","marker":"[24]"},{"why":"Is the representative prompt-tuning baseline whose frozen-parameter design the paper compares against full fine-tuning and questions.","marker":"[42]"},{"why":"Provides the code base and experimental protocol that Skip Tuning builds on for the main comparisons.","marker":"[39]"},{"why":"Is the leading prompt-tuning baseline for the headline base-to-new and domain generalization comparisons.","marker":"[18]"},{"why":"Represents the low-rank adapter baseline that Skip Tuning is compared with in the adapter table.","marker":"[15]"},{"why":"Represents the feature-adapter baseline that Skip Tuning is compared with in the adapter table.","marker":"[10]"}],"fun_headline_variants":["Skip Tuning: skip layers and class tokens for faster, better VLM adaptation","Layer-wise and class-wise skipping beat prompt tuning on VLMs","Cut layer and class tokens to fine-tune VLMs faster and better","Skip Tuning: adapt VLMs by skipping, not adding modules","Skip Tuning: no new parameters, just skip layers and class tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the importance rankings of layers and class tokens measured by Feature Sensitivity and Gradient Dependence on only two datasets (Caltech101 and Food101) hold across all 11 evaluation datasets and for unseen classes, so that one fixed skipped-layer count ($\\omega=6$) and one fixed class-sampling rule ($r=0.5$, $\\lambda=0.3$) work everywhere.","fun_headline_variants_meta":{"raw":{"variants":["Skip Tuning: skip layers and class tokens for faster, better VLM adaptation","Layer-wise and class-wise skipping beat prompt tuning on VLMs","Cut layer and class tokens to fine-tune VLMs faster and better","Skip Tuning: adapt VLMs by skipping, not adding modules","Skip Tuning: no new parameters, just skip layers and class tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001514,"raw_usage":{"total_tokens":6057,"prompt_tokens":922,"completion_tokens":5135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":5040}},"tokens_in":538,"tokens_out":5135,"duration_ms":27743,"temperature":1.0,"reasoning_tokens":5040,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:51:06.922487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure Feature Sensitivity on all 11 datasets before fixing $\\omega$; if any dataset shows large FS values in the first six layers, the LSkip premise fails there. A direct test is to run Skip Tuning on a dataset where shallow layers carry task-specific structure, such as a texture- or edge-dominated benchmark, and compare base and new accuracy against full fine-tuning; a large accuracy drop would falsify the claim that shallow layers are universally dispensable.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained CLIP model and the zero-shot image-text matching setup used as the testbed."},{"cited_title":"Dept: Decoupled prompt tuning","cited_arxiv_id":null,"evidence_quote":"Provides the code base and experimental protocol that Skip Tuning builds on for the main comparisons."},{"cited_title":"Self-regulating prompts: Foundational model adaptation without forgetting","cited_arxiv_id":null,"evidence_quote":"Is the leading prompt-tuning baseline for the headline base-to-new and domain generalization comparisons."},{"cited_title":"Clip-adapter: Better vision-language models with feature adapters","cited_arxiv_id":null,"evidence_quote":"Represents the feature-adapter baseline that Skip Tuning is compared with in the adapter table."}],"review_version":1}