{"id":"9f75e6c2-4db5-4d43-9fc4-e3455441fe86","arxiv_id":"2507.22872","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TR-PTS combines Fisher-information-based parameter selection with attention-based token pruning and merging to achieve state-of-the-art accuracy in efficient vision transformer fine-tuning.","lead":"The paper proposes TR-PTS, a method that fine-tunes only a small subset of a vision transformer's parameters, chosen by Fisher information, while also keeping only the most useful image tokens. The authors report top results on 24 visual classification tasks while updating under 1% of the model's parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sparse-insertion rule rests on a weakly-evidenced correlation between layer-wise parameter sparsity and token redundancy; if that correlation breaks on other tasks, token selection could discard informative tokens, undermining the claimed simultaneous accuracy and efficiency gains.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing point: the joint selection strategy depends on the empirical correlation claimed in Section 3.4 and Table 5. I agree with that assessment. The table's evidence is minimal (four datasets, two ratios, no variance), and the paper does not specify the token-selection hyperparameters used to produce the headline results, so the claimed efficiency–accuracy trade-off cannot be reproduced. My proposed test directly probes the universality of the sparsity–redundancy correlation by comparing Sparse, Dense, and Random placements across all VTAB-1k tasks. If the correlation fails on a substantial fraction of tasks, the method's central design choice is unsupported, and the reported gains over GPS would likely shrink or reverse. This supports the reader's CONDITIONAL verdict; no change to the verdict is needed beyond the conditions already stated.","tokens_in":13464,"tokens_out":9402,"duration_ms":105960,"concrete_test":"Run the full TR-PTS pipeline on all 19 VTAB-1k tasks with three token-placement variants: token reduction applied to (a) the layers with lowest FIM density ('Sparse'), (b) the layers with highest FIM density ('Dense'), and (c) randomly chosen layers, at selection ratio ρ=0.8, with three seeds each. Compute for each task the difference in test accuracy between Sparse and Dense placements and correlate it with the layer-wise FIM-density distribution. If Sparse is not at least as good as Dense on the majority of tasks (say ≥15/19), the claimed correlation is not universal and the unified strategy lacks the necessary support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.4, the paper motivates its central design choice—applying token reduction preferentially to parameter-sparse layers—by an 'empirical observation' of correlation between parameter sparsity and token redundancy. Table 5 is the only quantitative support. It covers just four of the 24 evaluated datasets (Sun397, Flower102, Dsprites/loc, Patch Camelyon) at two selection ratios (0.95, 0.8), and the differences between the 'Sparse' and 'Dense' placements are small (typically 0.1–1.0 accuracy points). No error bars, seeds, or a correlation coefficient are reported. The main experiments (Tables 2 and 3) do not state which layers receive token selection, how many layers are used, or the value of the selection ratio ρ, so the claim that TR-PTS 'reduces computational overhead' while improving accuracy cannot be independently verified. More importantly, if the sparsity–redundancy correlation is not consistent across tasks—e.g., a structured task where low-FIM layers still carry important spatial tokens—token selection at those layers could remove informative tokens, and the accuracy advantage would erode. Because this correlation is the basis for unifying parameter and token selection, it is load-bearing for the paper's central claim of a simultaneous accuracy and efficiency win.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TR-PTS, a fine-tuning method for Vision Transformers that combines Fisher-Information-based task-relevant parameter selection (with layer-wise allocation) and attention-based task-relevant token selection/merging. The two mechanisms are coordinated by applying token reduction preferentially to layers with sparse task-relevant parameters, based on an empirical correlation claimed in Section 3.4. The method is evaluated on FGVC (5 datasets) and VTAB-1k (19 datasets) using ViT-B/16 pretrained on ImageNet-21k, reporting state-of-the-art mean accuracy (75.92 on VTAB-1k, 91.94 on FGVC) while tuning only 0.34% and 0.60% of parameters, respectively, and reducing FLOPs, memory, and inference time.","tokens_in":13760,"tokens_out":5031,"duration_ms":56910,"significance":"If the reported results hold, TR-PTS is a practically useful contribution: it shows that task-aware parameter and token selection can be combined to reduce both training and inference cost while matching or exceeding full fine-tuning accuracy on 24 datasets. The paper is clearly written, the method is simple and well motivated, and the code is promised to be released. The two mechanisms are individually reasonable, and the benchmark coverage is broad. However, the claimed state-of-the-art advantage over GPS rests on small average margins (0.74 and 0.16 percentage points) without error bars or multiple seeds, and the core design rule for joint selection (sparse-insertion) is supported by only a four-dataset comparison with no quantitative correlation measure. The efficiency numbers also depend on unreported hyperparameters. These issues currently prevent the central simultaneous efficiency-accuracy claim from being fully verified, though the methodology is sound in principle.","major_comments":[{"comment":"The state-of-the-art claim over GPS rests on average margins of 0.74 percentage points on VTAB-1k and 0.16 percentage points on FGVC, with no error bars or multiple seeds. The FGVC margin in particular is smaller than typical seed-to-seed variation for this benchmark suite, so the improvement is not statistically supported as reported. Please report mean and standard deviation over at least three runs, or provide paired seed-matched comparisons against GPS, for the main tables.","section":"Section 4.2, Tables 2 and 3"},{"comment":"Key hyperparameters are not reported: M (top-Fisher percentage in Eq. (3)), C_min in Eq. (4), the token select rate ρ in Eq. (9), the number of samples used for the Fisher approximation in Eq. (2), and the specific layers at which token selection is applied. The main results in Tables 2 and 3 therefore cannot be reproduced or independently evaluated, and the accuracy-efficiency trade-off is not fully specified. Please add a hyperparameter table or state these values per benchmark.","section":"Sections 3.2-3.4 and 4.1"},{"comment":"The sparse-insertion rule is the load-bearing design choice that unifies parameter and token selection, but its empirical support is thin. Table 5 covers only four datasets that come from the same evaluation benchmarks, reports no error bars, and shows no consistent advantage for Sparse placement: at ρ=0.8, Sparse is equal to Dense on Flower102 (99.1 vs 99.1) and worse than Dense on Patch Camelyon (87.2 vs 87.4). The differences are typically 0.1-1.0 points, within the expected noise. To support the claimed correlation between parameter sparsity and token redundancy, please provide a quantitative per-layer analysis (e.g., correlation between FIM-based parameter sparsity and attention entropy or token redundancy) over a wider set of tasks, or validate the sparse-insertion rule on held-out tasks not used to motivate it.","section":"Section 3.4, Table 5"},{"comment":"The computational cost claims (shortest inference time, lowest memory, lowest FLOPs) are difficult to verify because the paper does not state the token schedule used in the efficiency measurements: which layers perform token selection, what ρ is used, and what the resulting average token counts are. Inference time is also reported without variance despite being averaged over only 500 forward passes. Please specify the configuration behind Figure 4 so the reported efficiency numbers can be reproduced.","section":"Section 4.2, Figure 4"}],"minor_comments":[{"comment":"The abstract contains grammatical errors: 'benchmark' should be 'benchmarks', and 'The code are available' should be 'The code is available'.","section":"Abstract"},{"comment":"In the FGVC results description, 'slightly surpassing GPS (91.78%) and and outperforming full fine-tuning' contains a duplicated 'and'.","section":"Section 1"},{"comment":"The heading 'Task-Relevant T oken Selection' contains a stray space in 'Token'.","section":"Section 3.3"},{"comment":"The caption reads 'Differernt Datasets Top 1% FIM Parameter Distribution'; 'Differernt' should be 'Different'.","section":"Figure 5 caption"},{"comment":"The formula Cl = max(1, wl/min(w) * Cmin) is undefined if the least important layer has wl = 0, which would make min(w) = 0. Please clarify how zero-importance layers are handled, since the text states that every layer retains at least one active connection.","section":"Section 3.2.2, Eq. (4)"},{"comment":"The terms 'Dense', 'Sparse', and 'Random' are not quantitatively defined. Please state the criterion used to classify layers as dense or sparse (e.g., top/bottom fraction by selected-connection density) and how 'Random' placement is sampled.","section":"Table 5"},{"comment":"The sentence 'pre-train the model with both supervised method' should be rephrased, e.g., 'pre-trained with supervised methods' or 'supervised pretraining'.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is a reasonable combination of two existing lines (Fisher-based parameter selection and attention-based token merging), with a coordination rule that is the main novelty. The central risk is the thin empirical support for that coordination rule and the absence of statistical confidence for the small margins over GPS. The authors should be asked to add error bars or seed-matched comparisons, report all hyperparameters, and strengthen the validation of the sparse-insertion correlation. This is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is accurate: this is a solid, incremental PEFT paper. The new thing is the joint formulation – FIM-based layer-wise parameter selection (close to GPS) combined with attention-based token selection and merging (close to ToMe) – plus the sparse-insertion rule that applies token reduction preferentially to parameter-sparse layers. The writing is clear, the method is sensible, and the experiments cover 24 datasets with a sensible baseline set. There is a code link, which is good practice, and the ablations (Table 4) show both components help.\n\nThat said, the soft spots are real and need to be addressed before I'd trust the headline numbers. First, the claimed SOTA margins over GPS are thin: +0.74% on VTAB-1k and +0.16% on FGVC. There are no error bars, no multiple seeds, and no hyperparameter settings reported (M, C_min, rho, Fisher sample size, which layers get token selection). That makes it impossible to distinguish a real improvement from noise. Second, the sparse-insertion rule is load-bearing: the paper's central efficiency-accuracy win depends on the claim that layers with fewer task-relevant parameters also encode less informative tokens. The support for this is just Table 5, which covers 4 of the 24 datasets, reports differences of 0.1–1.0 accuracy points, and shows no variance or correlation coefficient. I think the stress-test note is correct to flag this as a weakly evidenced correlation. If it fails on other tasks, applying token selection at sparse layers could discard informative tokens, and the method's advantage could erode.\n\nThe reader's concern about circularity is minor, and I agree it doesn't force the central result. The placement rule was tuned on the benchmark, but accuracy is still measured on held-out test sets. That's a small burden, not a fatal one.\n\nOverall: this is a competent contribution that deserves a serious referee, but the revision needs multi-seed results with variance, full hyperparameter reporting, and a more thorough validation of the sparse-insertion heuristic, ideally across a broader set of tasks or with a quantitative measure of the sparsity–redundancy correlation. I would not desk-reject it, but I would not accept it in this form either.","headline":"A competent, incremental PEFT paper whose joint parameter-and-token selection is plausible but whose central sparse-insertion claim rests on thin evidence; referee it, but ask for statistical rigor and full hyperparameter disclosure.","tokens_in":14280,"tokens_out":1713,"would_cite":false,"duration_ms":22189,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A unified task-aware scheme that selects which parameters to unfreeze and which image tokens to keep can beat full fine-tuning of a Vision Transformer while updating less than one percent of its weights.","keywords":["TR-PTS","parameter-efficient fine-tuning","Fisher information matrix","token selection","token merging","Vision Transformer","VTAB-1k","FGVC"],"falsifier":"A concrete test would be to run TR-PTS with token reduction placed in dense, sparse, and random layers across many tasks and look for a task where sparse placement underperforms dense or random; the paper's Table 5 shows only four datasets. A second test would measure the sparsity-redundancy correlation directly by computing layer-wise Fisher scores and token-importance distributions for a new task and checking whether they are actually negatively correlated before applying the method.","tokens_in":13273,"feed_emoji":"🎯","tokens_out":9082,"duration_ms":97767,"temperature":0.7,"pith_summary":"TR-PTS is a method for adapting a pre-trained Vision Transformer to a new visual task that decides, before training, which parameters to unfreeze and, during training, which image tokens to keep. The parameter side ranks weights by their Fisher information, approximated as the squared gradient of the loss, and allocates trainable connections per layer in proportion to task relevance. The token side uses the [CLS] token's attention scores to keep the most informative patches and merges the rest into one weighted token. The two selections are coordinated: token reduction is applied mostly in layers that were given fewer trainable parameters, based on the paper's observation that such layers also carry less informative tokens. On the FGVC and VTAB-1k benchmarks, the method reports average accuracies that exceed full fine-tuning by 3.40 and 10.35 percentage points while updating only 0.60% and 0.34% of the pre-trained model's parameters.","feed_headline":"Tuning 0.6% of parameters beats full fine-tuning","feed_subtitle":"Fisher-information layer scores and CLS-attention token merging beat full fine-tuning on 24 vision tasks.","key_machinery":"The central machinery is a Fisher-information-guided binary mask on the weights, paired with a [CLS]-attention token selector and merger. The Fisher Information Matrix is approximated by the squared gradient of the cross-entropy loss; its diagonal gives each parameter a task-relevance score. Layer importance is computed as the share of the top-M% parameters that fall in each layer, and each neuron in a layer is assigned $C_l$ trainable connections via $C_l = \\max(1, w_l/\\min(w) \\cdot C_{\\min})$. Token selection uses the attention score $a_i = \\frac{\\exp(q_{\\mathrm{CLS}}\\cdot k_i)}{\\sum_j \\exp(q_{\\mathrm{CLS}}\\cdot k_j)}$ for each patch token, keeping the top $\\lfloor \\rho N \\rfloor$ tokens and fusing the rest into $x_{\\mathrm{merged}} = \\sum_{i\\in I} a_i x_i / \\sum_{i\\in I} a_i$. The two are joined by a \"sparse insertion\" rule: token reduction is applied at layers with few trainable connections, which the paper's Table 5 reports is more accurate than applying it to dense or randomly chosen layers.","core_discovery":"The paper claims that parameter and token selection for fine-tuning should be task-driven and coupled, not fixed and independent. It introduces a layer-wise Fisher Information Matrix score to find the connections most sensitive to the target task, guarantees every layer keeps at least one active connection per neuron, and scales the number of trainable connections per layer by its task importance. In parallel, it scores image tokens by their attention weight from the [CLS] token, keeps the top fraction, and merges the discarded tokens into a single weighted token so no information is wholly lost. The coordinated rule is that token reduction happens preferentially in parameter-sparse layers, where the paper argues tokens are less informative. With both mechanisms active, TR-PTS reports average Top-1 accuracy of 91.94% on FGVC and 75.92% on VTAB-1k, beating full fine-tuning, GPS, SSF, LoRA, and other baselines while adding no extra parameters at inference time.","pith_inferences":["If the layerwise sparsity-redundancy correlation holds beyond the four datasets shown in Table 5, the same coordination rule could be ported to other transformer families and modalities by setting each layer's token-reduction rate as a decreasing function of its parameter-importance score.","The Fisher mask is computed once at the start; a testable variant would recompute it after token selection starts changing the input distribution, since the most task-relevant parameters may shift as the token stream is refined.","The reported VTAB gain over full fine-tuning is not yet isolated from the token-selection effect: comparing full fine-tuning with the same token selection and merging would tell how much of the 10.35-point gain comes from parameter selection alone.","Cheaper importance signals such as gradient norms or attention entropy could replace the Fisher scores; the paper argues for Fisher over gradient magnitude but does not include those alternatives in its comparisons."],"forward_implications":["Fine-tuning a ViT-B/16 with 0.34% of its parameters and a shortened token sequence can exceed full fine-tuning by 10.35 mean Top-1 points on the 19 tasks of VTAB-1k.","On the five FGVC tasks, the same recipe exceeds full fine-tuning by 3.40 points while updating 0.60% of parameters, with the largest per-task gains on Stanford Dogs and NABirds.","Because TR-PTS adds no modules and merges discarded tokens into a single weighted token, the deployed model carries no extra parameters and performs fewer FLOPs and less per-sample memory than the compared PEFT baselines.","Token reduction should be applied in parameter-sparse layers rather than dense ones; the paper's comparisons show that \"dense\" token selection can hurt accuracy while \"sparse\" selection improves it.","Task-driven parameter sets overlap little across tasks, so the Fisher-based allocation selects genuinely task-specific connections rather than a fixed sparse mask."],"supporting_citations":[{"why":"Defines the Fisher Information Matrix that TR-PTS uses to score each parameter's task relevance.","marker":"[16]"},{"why":"Provides the gradient-based per-neuron connection allocation scheme that TR-PTS adapts and the strong partial-tuning baseline it compares against in Tables 2 and 3.","marker":"[39]"},{"why":"Supplies the full fine-tuning protocol and benchmark task definitions on FGVC and VTAB-1k whose accuracies TR-PTS claims to surpass.","marker":"[14]"},{"why":"LoRA is a reparameterization baseline included in the accuracy and efficiency comparisons.","marker":"[12]"},{"why":"BitFit is a bias-only partial-tuning baseline included in the comparisons.","marker":"[36]"},{"why":"Supplies the token-merging idea that TR-PTS adapts, fusing discarded tokens by weighted averaging instead of dropping them.","marker":"[1]"},{"why":"Provides the dynamic token sparsification precedent that motivates the token-reduction component of TR-PTS.","marker":"[25]"},{"why":"Defines the VTAB-1k benchmark tasks and evaluation setup on which TR-PTS reports its average accuracy.","marker":"[37]"}],"fun_headline_variants":["Task-driven parameter and token selection beats full fine-tuning","Fisher-info parameter picking and token merging top full fine-tuning","Selective fine-tuning with Fisher scores boosts accuracy beyond full tuning","0.6% trainable parameters outperform full fine-tuning on vision tasks","TR-PTS: task-aware parameter and token selection for efficient tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The joint strategy assumes that layers whose parameters look task-irrelevant under the Fisher score also carry tokens that are safe to drop, so applying token reduction to sparse layers preserves accuracy; if that correlation fails for a given task, informative tokens could be discarded and the accuracy gains would shrink.","fun_headline_variants_meta":{"raw":{"variants":["Task-driven parameter and token selection beats full fine-tuning","Fisher-info parameter picking and token merging top full fine-tuning","Selective fine-tuning with Fisher scores boosts accuracy beyond full tuning","0.6% trainable parameters outperform full fine-tuning on vision tasks","TR-PTS: task-aware parameter and token selection for efficient tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1325,"prompt_tokens":969,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":585,"tokens_out":356,"duration_ms":4884,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:11:16.113788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to run TR-PTS with token reduction placed in dense, sparse, and random layers across many tasks and look for a task where sparse placement underperforms dense or random; the paper's Table 5 shows only four datasets. A second test would measure the sparsity-redundancy correlation directly by computing layer-wise Fisher scores and token-importance distributions for a new task and checking whether they are actually negatively correlated before applying the method.","supporting_citations":[{"cited_title":"Overcoming catastrophic forgetting in neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the Fisher Information Matrix that TR-PTS uses to score each parameter's task relevance."},{"cited_title":"Gradient-based parameter selection for efficient fine-tuning","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-based per-neuron connection allocation scheme that TR-PTS adapts and the strong partial-tuning baseline it compares against in Tables 2 and 3."},{"cited_title":"Visual prompt tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the full fine-tuning protocol and benchmark task definitions on FGVC and VTAB-1k whose accuracies TR-PTS claims to surpass."},{"cited_title":"Lora: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA is a reparameterization baseline included in the accuracy and efficiency comparisons."},{"cited_title":"Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models","cited_arxiv_id":null,"evidence_quote":"BitFit is a bias-only partial-tuning baseline included in the comparisons."},{"cited_title":"Token merging: Your vit but faster, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the token-merging idea that TR-PTS adapts, fusing discarded tokens by weighted averaging instead of dropping them."},{"cited_title":"Dynamicvit: Efficient vision trans- formers with dynamic token sparsification","cited_arxiv_id":null,"evidence_quote":"Provides the dynamic token sparsification precedent that motivates the token-reduction component of TR-PTS."}],"review_version":1}