{"id":"078ba486-37fc-4355-94f7-023fcdacd57f","arxiv_id":"2608.11758","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AWARe freezes the highest-activation neurons in selected projection layers during fine-tuning, reducing catastrophic forgetting in multimodal LLMs.","lead":"This paper presents AWARe, a fine-tuning method that computes a neuron importance score from activation patterns on a small calibration set and freezes the most important neurons while training on a new task. The method requires no architecture changes and is claimed to preserve upstream capabilities better than existing anti-forgetting techniques for multimodal language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline retention numbers are selected by tuning the retention ratio on the evaluation benchmark; without an independent validation split, the claim that 30% is 'often sufficient' is not yet supported.","rationale":"The single most load-bearing condition for the central claim is not the existence of a good mask per se—Table 3 shows activation selection beats random selection, which is real evidence—but that the specific numbers used to support 'often sufficient' are produced by a configuration that was tuned on the evaluation benchmark. The paper presents Table 4 as an ablation, but it is also the selection mechanism for the headline result: the authors choose the Global-Highest 30% row because it maximizes H. Since H is the headline metric, optimizing H on IconQA is optimizing the reported outcome. The same critique applies to the target-layer set and selection strategy. This is not a claim of fraud; it is a standard selection-on-test problem. The sensitivity analysis in Table 6 only varies calibration sampling, not the hyperparameters that carry the result. A nested validation check would settle it cleanly: if 30% global-highest is also selected on validation and the held-out H is stable, the central claim is supported; if not, the method's advantage may be smaller than reported. This is why the verdict should remain conditional pending that check.","tokens_in":19172,"tokens_out":12776,"duration_ms":143933,"concrete_test":"Use a nested validation protocol before touching the test benchmarks: split IconQA (and COCO-Caption) into a validation subset and a held-out test subset, or use a separate set of downstream tasks. Select ρ ∈ {10%, 30%, 50%}, selection strategy (global vs layer-balanced), and target components on the validation subset only; then freeze the chosen configuration and report R/E/H on the held-out subset, for both upstream-task calibration and MMMU calibration. If the selected ρ is not 30%, or if the held-out H is more than ~2 points below the Table 1 values, the reported numbers are partly an artifact of test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline trade-off (R=98.4, H=103.2 on IconQA) is obtained after sweeping the retention ratio ρ from 1% to 90% and selecting the value that maximizes H on the same evaluation benchmarks (Table 4; Eq. 6). Table 11 then reports the hyperparameters for this 'optimal' configuration, and Section D states all results are averaged over three runs but does not describe any held-out validation split. Because H is computed from the upstream retention R and downstream target accuracy, choosing ρ by H on the final benchmarks is equivalent to selecting hyperparameters on the test set. The same issue applies to the choice of global-highest selection and to the target set q/k/v + mm_projector, which are also ablated on IconQA. The MMMU-calibration row is more robust to data availability but still uses the ρ and target set selected on IconQA. Therefore the paper's central claim that freezing the top 30% is 'often sufficient' is a post-selection statement, not a tested prediction. If a validation-based protocol changes the optimal ρ or lowers H materially, the central quantitative claim is overstated; if it does not, the concern is resolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AWARe, a fine-tuning method for multimodal LLMs that estimates per-neuron activation saliency on a small calibration set, then freezes the top ρ fraction of output rows in selected q/k/v and mm_projector linear layers during downstream training while updating the remaining rows. The method is evaluated on LLaVA-v1.5-7B for single-task adaptation on IconQA and COCO-Caption, on MLLM-DCL for continual instruction tuning, and on Qwen2.5-VL for two downstream tasks. The central claims are that activation-based selective freezing preserves upstream knowledge with only a modest loss of downstream plasticity, that freezing roughly 30% of the salient attention parameters is often sufficient, and that a general-purpose calibration set such as MMMU can substitute for unavailable upstream data.","tokens_in":19312,"tokens_out":8436,"duration_ms":81895,"significance":"If the empirical claims hold, AWARe is a useful addition to the anti-forgetting toolbox for MLLMs: it is architecture-agnostic, requires no replay buffer or extra adapter modules, and its saliency score is a deterministic function of activations rather than a fitted model. The ablations in Table 3 give direct evidence that activation-based selection beats random and weight-norm selection, and the MMMU-calibration results in Table 1 address a practically important data-availability scenario. The core equations are simple enough to be checked and reimplemented, and a code link is provided. The main weaknesses are methodological: the headline hyperparameters are selected on the evaluation benchmarks, and the main tables lack variance estimates, so the strength of the comparative claims is currently not fully supported.","major_comments":[{"comment":"The headline stability-plasticity numbers are selected on the evaluation benchmarks rather than on an independent validation split. In Table 4, ρ is swept from 1% to 90% and the value maximizing H (Eq. 6) on IconQA is reported; Table 11 then lists the resulting 30% global-highest configuration as optimal, and Section D states only that results are averaged over three runs. Because R and E are computed from upstream retention and downstream target accuracy on the same test benchmarks, this is hyperparameter selection on the test set. The same issue applies to the choice of target set (q/k/v + mm_projector), which is ablated on IconQA in §4.5/Fig. 5 and then transferred to COCO-Caption, MLLM-DCL, and Qwen2.5-VL. Consequently, the abstract's claim that 'freezing the top 30%... is often sufficient' is a post-selection observation, not a tested prediction. Please add a validation-based selection protocol (e.g., hold out part of the downstream training set or a separate task for choosing ρ and the target set), or clearly label the reported numbers as selected-on-test and show that the choice of ρ is not material to the conclusions.","section":"§4.5, Table 4; §D, Table 11; Eq. (6)"},{"comment":"Equation (8) defines E relative to full fine-tuning and the text calls Accdown,Full-FT the 'performance upper bound achievable by Full-FT', but Table 1 contradicts this: on COCO-Caption, LoRA reaches 110.27, Model Tailor 117.64, LoRASculpt 120.35, and AWARe 120.94, all above the Full-FT value of 95.59, giving E>100 in several rows. The bounded interpretation of E, and hence of H as a harmonic mean of stability and plasticity, is therefore not valid as stated. Please replace the 'upper bound' wording or redefine E, and check whether any conclusions change under an alternative normalization.","section":"§4.3, Eq. (8); Table 1"},{"comment":"Section D reports that all results are averaged over three random runs, but Tables 1 and 2 report only point estimates, and Table 6 reports standard deviations only for the IconQA calibration-size sensitivity analysis. The reported improvements, such as +3.1 in H over LoRASculpt on IconQA and +3.18 in Avg over DISCO on MLLM-DCL, cannot be assessed for statistical significance. Please report mean ± std (or confidence intervals) for the main tables, at least for AWARe and the closest baselines, or provide per-run tables in the appendix.","section":"§4, Tables 1–2; §D"}],"minor_comments":[{"comment":"The text says the L2-norm is computed 'for each sample j and neuron k', but the equation uses the index i for the sample; please make the indexing consistent.","section":"§3.2, Eq. (1)"},{"comment":"The text states that the method 'updates only a quarter of the model parameters', but Eq. (14) gives a trainable ratio of about 17.5%. Please correct the wording to match the calculation, and note that the calculation excludes the mm_projector parameters.","section":"§C, Eq. (14)"},{"comment":"The 30% global-highest configuration is called optimal with H=103.2, but the 40% row also attains H=103.2; please acknowledge the tie or explain the additional criterion used to select 30%.","section":"§4.5, Table 4"},{"comment":"The cited finding that 'updating self-attention projections tends to cause significantly less catastrophic forgetting' than MLP blocks is not obviously the rationale for freezing self-attention rows; please clarify the logical connection between the citation and the design choice.","section":"§3.4"},{"comment":"Table 11 specifies a calibration composition of '200 samples each' from four upstream tasks (800 total), while Section 4.6 describes calibration sizes of 200 and 400 samples; please clarify whether these are per-task or total-sample counts.","section":"§4.6 and Table 11"}],"recommendation":"major_revision","confidential_remarks":"The core risk is not circularity or a flawed mechanism, but the fact that the headline hyperparameters (ρ and the target set) are selected on the evaluation benchmarks and then stated as general conclusions. A revision that adds a validation-based selection protocol and variance reporting would resolve the main concern. If the authors are unable to add a validation split, the empirical claims should be rewritten as descriptive post-selection results rather than predictive statements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick note on AWARe (arXiv:2608.11758).\n\nWhat you should know: this is a straightforward, well-executed empirical paper. It takes the activation-based saliency idea from AWQ/Wanda and applies it to selective parameter freezing during fine-tuning of MLLMs to reduce catastrophic forgetting. The method is simple: compute per-neuron activation magnitudes on a small calibration set, freeze the top-ρ% rows of q/k/v and the mm_projector, train the rest. Ablations actually support the design: activation selection beats random and weight-norm, and MMMU works as a calibration proxy when upstream data is unavailable. The Qwen2.5-VL results are a nice generality check. I believe the reported stability-plasticity improvements are real.\n\nThe soft spot is the one the stress-test caught: ρ is swept from 1% to 90% on IconQA and the value that maximizes H on that same benchmark is then presented as the \"optimal\" configuration. There is no held-out split. So the claim that \"freezing the top 30% is often sufficient\" is a post-selection statement, not a tested prediction. It is not fatal — Table 4 shows a plateau (10% gives H=101.3, 40% ties 103.2), and 30% transfers without re-tuning to COCO-Caption and MLLM-DCL — but it should be fixed by reporting a validation-based protocol. The other issues are minor: no error bars in Tables 1-2 (only in the calibration sensitivity), baseline hyperparameters unspecified, and the Section C parameter-efficiency calculation silently ignores the mm_projector.\n\nThe paper is honest about its limitations, the math is clean, and there is no circularity beyond the standard hyperparameter tuning. It is a useful engineering contribution, not a paradigm shift. Send it to peer review, but ask for error bars, a validation split for ρ, and fuller baseline details. If those are fixed, it would make a solid baseline for anyone working on forgetting in MLLMs.","headline":"A clean, if modest, empirical paper that reuses activation saliency from pruning/quantization to freeze important rows during fine-tuning; the headline numbers are real, but the optimal 30% freeze ratio is tuned on the test set, so the central claim is partly post-selection.","tokens_in":19921,"tokens_out":4094,"would_cite":true,"duration_ms":40679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AWARe claims that freezing the output neurons with the strongest upstream activations, about 30 percent of self-attention projections, keeps a multimodal model's prior knowledge nearly intact during downstream fine-tuning.","keywords":["catastrophic forgetting","multimodal large language models","activation-based saliency","selective freezing","continual instruction tuning","parameter-efficient fine-tuning","knowledge retention","plasticity-stability trade-off"],"falsifier":"Measure the activation vectors of the frozen neurons on the calibration set before and after downstream fine-tuning. If those activations drift substantially while upstream retention stays high, the mask is not what is pinning the retained pathway, and the reported retention would have to come from some other part of the training recipe.","tokens_in":18915,"feed_emoji":"🧠","tokens_out":14421,"duration_ms":128822,"temperature":0.7,"pith_summary":"The paper is trying to solve catastrophic forgetting in multimodal large language models: when a model that already understands images and text is fine-tuned on one new task, it tends to lose its earlier general capabilities. AWARe claims that a small set of output neurons, identified by their activation strength on upstream data, is disproportionately responsible for holding that prior knowledge, and that freezing the top 30% of those neurons during fine-tuning preserves upstream ability while leaving the model enough freedom to learn the new task. The method needs only a forward pass on a few hundred calibration samples, changes no architecture, and can even use a general-purpose multimodal benchmark when upstream data are unavailable. The reported numbers on IconQA, COCO-Caption, and the MLLM-DCL sequence support that claim, with retention $R=98.4$ and harmonic means above every listed baseline. If correct, this gives practitioners a cheap, deployment-compatible way to keep a multimodal model general while specializing it.","feed_headline":"Freezing 30% of attention neurons stops multimodal forgetting","feed_subtitle":"AWARe reports 98.4% upstream retention on IconQA while still beating full fine-tuning on the new task.","key_machinery":"The central object is an activation-weighted saliency score computed per output neuron. Given a target linear layer's activation tensor $A \\in \\mathbb{R}^{B \\times L \\times d_{\\mathrm{out}}}$, the method first aggregates activations along the sequence length with the $L^2$ norm, then normalizes each sample's vector across the hidden dimension, and finally averages over the batch: $s_k = \\frac{1}{B} \\sum_{i=1}^B \\frac{\\sqrt{\\sum_{j=1}^L a_{i,j,k}^2}}{\\|\\mathbf{a}'_i\\|_2}$. Neurons are ranked by $s_k$ globally across the target layers, and the top $\\rho$ fraction (default $\\rho = 0.3$) are frozen by setting the corresponding rows of the gradient mask to zero, while the remaining rows receive the full gradient update. The mask is applied only to the self-attention projections $q\\_proj$, $k\\_proj$, $v\\_proj$ and to the multimodal projector $mm\\_projector$, with all other components kept frozen. What carries the argument is the claim, supported by the paper's ablations, that this particular score locates the output dimensions whose preservation is sufficient for upstream knowledge, whereas random or weight-magnitude selection at the same ratio loses it.","core_discovery":"On the paper's own terms, AWARe's central claim is that you can mitigate catastrophic forgetting in a multimodal large language model by computing a per-neuron saliency score from forward activations on a small calibration set and freezing the top-ranked output neurons in the self-attention q/k/v projections and the multimodal projector during downstream fine-tuning. This is a selective-freezing rule, not a new architecture and not a replay strategy. The paper reports that with the top 30% of globally salient neurons frozen, LLaVA-v1.5-7B fine-tuned on IconQA keeps upstream retention $R=98.4$ and reaches a harmonic mean $H=103.2$; on COCO-Caption it reaches $H=108.0$; and on the five-task MLLM-DCL continual learning benchmark it reaches an average of $65.58$, with all numbers above the listed baselines. It further claims the same scheme remains effective when the calibration set is a general-purpose benchmark such as MMMU instead of the upstream task data.","pith_inferences":["If high-activation output neurons are where stable knowledge lives, the same saliency score could be reused for model merging or pruning decisions, since it is obtained in one forward pass and requires no labels; the paper does not test this.","The paper's layer ablation suggests a modular map: self-attention projections tolerate task-specific updates while MLP blocks are so knowledge-dense that even protected fine-tuning collapses upstream accuracy. One could test that map directly by comparing per-layer freezing masks across different base models and task families.","Because the paper evaluates one seven-billion-parameter model and a five-task sequence, a natural extension is to check whether the saliency ordering stays stable over much longer continual learning streams; repeated updates could shift activations so that the original frozen mask becomes stale."],"forward_implications":["Freezing the globally top 30% of salient q/k/v and mm_projector neurons is enough for near-complete upstream retention: on IconQA the paper reports $R=98.4$ with target accuracy 86.92, above the full fine-tuning reference of 80.15.","The method transfers without upstream data: using a general-purpose benchmark as the calibration set keeps $H=101.7$ on IconQA and $H=107.9$ on COCO-Caption.","In a five-task continual instruction tuning sequence (MLLM-DCL), AWARe reports average and last-task scores of 65.58 and 62.10, both above the listed baselines.","Because freezing happens at the neuron-row level, only about 17.5% of the model's parameters are updated in the optimal setting, and the training-time wrapper can be removed after fine-tuning so inference remains standard.","Ablations show the choice of saliency signal is load-bearing: random selection of the same 30% of neurons drops average upstream accuracy to 49.94 on IconQA, while activation-based selection keeps it at 60.13."],"supporting_citations":[{"why":"Shows that a small fraction of weights associated with large activation magnitudes carry a disproportionate share of model behavior under quantization, motivating the activation-based saliency approach.","marker":"(Lin et al., 2024)"},{"why":"Provides the complementary pruning result that combining weight magnitude with input activation statistics estimates functional importance better than weights alone.","marker":"(Sun et al., 2024)"},{"why":"Supplies the finding that updating self-attention projections causes less catastrophic forgetting than updating MLP blocks, which justifies restricting AWARe to q/k/v and mm_projector.","marker":"(Zhu et al., 2025b)"},{"why":"Provides the MMMU general-purpose benchmark used as a calibration set when upstream data are unavailable.","marker":"(Yue et al., 2024)"},{"why":"Defines the MLLM-DCL continuous instruction tuning benchmark and its evaluation metrics, which AWARe uses for the continual learning experiments.","marker":"(Zhao et al., 2025)"},{"why":"Defines the LoRA baseline that AWARe compares against and whose low-rank updates are the main parameter-efficient alternative.","marker":"(Hu et al., 2022)"},{"why":"Defines LoRASculpt, the strongest single-task retention baseline in the paper's comparison tables.","marker":"(Liang et al., 2025)"},{"why":"Defines SPIDER, another saliency-based selective tuning baseline that the paper must beat in both single-task and continual settings.","marker":"(Huang et al., 2025)"}],"fun_headline_variants":["AWARe: Freeze key neurons, forget less in multimodal models","Activation-based freezing curbs catastrophic forgetting in MLLMs","Selective neuron freezing hits 98.4% upstream retention","AWARe: Adaptive retention stops forgetting without new architecture","Targeted neuron freeze preserves multimodal knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the neurons which fire most strongly on upstream data are exactly the ones whose preservation keeps upstream knowledge intact; if a different model or task distribution stores prior knowledge elsewhere, the frozen mask protects nothing.","fun_headline_variants_meta":{"raw":{"variants":["AWARe: Freeze key neurons, forget less in multimodal models","Activation-based freezing curbs catastrophic forgetting in MLLMs","Selective neuron freezing hits 98.4% upstream retention","AWARe: Adaptive retention stops forgetting without new architecture","Targeted neuron freeze preserves multimodal knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2439,"prompt_tokens":941,"completion_tokens":1498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1416}},"tokens_in":557,"tokens_out":1498,"duration_ms":11142,"temperature":1.0,"reasoning_tokens":1416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:28:10.891373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the activation vectors of the frozen neurons on the calibration set before and after downstream fine-tuning. If those activations drift substantially while upstream retention stays high, the mask is not what is pinning the retained pathway, and the reported retention would have to come from some other part of the training recipe.","supporting_citations":[],"review_version":1}