{"id":"7c2621a1-a99b-4c50-8a65-97bdf24f5c05","arxiv_id":"2504.13407","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Freezing the most important ViT parameter matrices before each new task, on top of orthogonal LoRA composition, reduces forgetting and improves average accuracy in class-incremental learning.","lead":"This paper proposes a way to make continual learning with pre-trained vision models forget less, by freezing the parts of the model that matter most for old tasks while learning new tasks. The method, called LoRAC-IPC, combines orthogonal low-rank adapters with importance-based freezing and reports state-of-the-art accuracy on several image classification benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3's global omega update can silently modify the very matrices IPC claims to freeze, leaving the central mechanism unspecified.","rationale":"The reader's weakest_assumption already targeted the interaction between globally updated omega coefficients and frozen matrices. My reading of Eq. (1), Section 3.3, and Eq. (11) confirms that the paper never specifies a masking rule for frozen W matrices in the loss or the optimization. Because the paper's main empirical claims (6.35% higher accuracy, 3.24% lower forgetting) and its Fig. 5 evidence for IPC are attributed to freezing, and because the code is publicly available, this is the single most load-bearing concern: if the implementation does not match the equations, the mechanism is ambiguous; if it does match only via an undocumented mask, the equations need correction. This does not invalidate the method, but it does mean the central claim is not fully established from the written text alone. A single targeted experiment—instrumenting whether frozen matrices actually change under the released update rules—would resolve the concern. I therefore keep the reader's CONDITIONAL verdict rather than moving to rejection, since the concern is checkable and may already be resolved in code.","tokens_in":19191,"tokens_out":3893,"duration_ms":40029,"concrete_test":"Using the released code (https://github.com/learninginvision/LoRAC-IPC) on Sup-21K Split CIFAR-100, train task 1, compute S(W) per matrix, and freeze the top-p matrices as specified. Then instrument training of task 2: record the Frobenius norm of the change in each frozen composed matrix after the first optimizer step and after the full task. If any frozen matrix changes by more than machine precision, the freeze is not enforced as stated. As a second check, run the task-2 update with all ω_i associated with frozen matrices also frozen (or zeroed) and compare against the reported LoRAC-IPC result in Table 5; if accuracy or forgetting moves by more than the reported std, the current headline result is entangled with an unspecified update mechanism.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim depends on the mechanism that top-p critical parameter matrices selected after task t remain unchanged during later tasks. The paper's equations do not actually implement this. In Eq. (1), every W matrix is W0 + Σ_i ω_i ΔW_i for all LoRA modules, and Section 3.3 states that during task t+1 “we also update exclusively the weight coefficients ω1,ω2,...,ωt with a low learning rate to slightly relax orthogonality.” If these ω are updated globally, then for any matrix W_m previously frozen by IPC, W_m^{t+1} changes through the terms ω_i ΔW_i even if the new ΔW_{t+1} were somehow prevented from entering W_m. Moreover, Eq. (11) and the surrounding text do not include any masking or zeroing of ΔW_{t+1} for frozen matrices. Thus the total loss minimizes over Θ_{t+1} while the stated constraint is only asserted in prose, not expressed in the objective or update rules. This is not a cosmetic ambiguity: the headline gains (6.35% accuracy, 3.24% forgetting on Split CIFAR-100) and the Fig. 5 evidence for IPC all depend on which matrices actually change during later tasks. If frozen matrices do change because ω updates are global, then the claimed “important parameter constraints” are not operative as described, and the improvement attributed to IPC may instead come from the low-learning-rate ω relaxation, the orthogonal loss, or task-ID inference. Conversely, if the released code masks omega updates and new LoRA modules for frozen matrices, then Eq. (1) and the Section 3.3 sentence misdescribe the method. Either way, the paper needs to pin down the update rule precisely before the central mechanism can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoRAC-IPC, a rehearsal-free continual learning method for pre-trained Vision Transformers. It combines sequentially learned LoRA modules in a weighted composition, imposes an orthogonality regularization on the QR-decomposed projection matrices, and proposes to freeze the most important parameter matrices (selected by a smoothed sensitivity–uncertainty score) before learning subsequent tasks. It also introduces a task-ID inference scheme and a task-adaptive classifier adjustment. On Split CIFAR-100, Split ImageNet-R, 5-datasets, Split DomainNet, and two multimodal datasets, the method reports state-of-the-art or competitive accuracy and forgetting.","tokens_in":19543,"tokens_out":7769,"duration_ms":69677,"significance":"If the mechanism is correctly specified, LoRAC-IPC is a simple and strong rehearsal-free baseline. The empirical study is extensive: three PTMs, four standard benchmarks, two multimodal datasets, ablations of rank, trade-off lambda, and component analysis. The paper provides code, and the central observation that orthogonal LoRA tuning does not prevent changes in important parameters is interesting and potentially useful. However, the description of the freezing mechanism is internally ambiguous, and the current text does not guarantee that the claimed constraints are operative as stated.","major_comments":[{"comment":"The interaction between the IPC freezing and the global update of the omega coefficients is unspecified. Equation (1) defines every parameter matrix W as a sum over all LoRA modules scaled by omega. In task t+1, Section 3.3 states that omega_1,...,omega_t are updated with a low learning rate. Under Eq. (1), this update changes every previously frozen matrix unless an explicit mask or snapshot is used. Equation (11), the total training loss, contains no masking, stop-gradient, or constraint term that would prevent the new LoRA module Delta W_{t+1} from contributing to the frozen matrices, nor does it express the freezing as a constraint on the optimization. The authors must specify the exact update rules: are frozen matrices snapshotted and excluded from the composition for subsequent tasks, is there a per-matrix stop-gradient on the omega coefficients, or is the new LoRA module added only to non-frozen matrices? This is load-bearing because the central claim and the IPC versus Random result in Table 4 are attributed to the freezing mechanism.","section":"Section 3.3, Eq. (1), Eq. (11)"},{"comment":"The importance score is defined for a trainable parameter w_{t,ij}, but Eq. (9) averages it over the composed matrix W_t, whose elements are not themselves trainable parameters (they are sums of pre-trained weights and LoRA updates). The manuscript does not state whether the gradient in Eq. (6) is taken with respect to the composed matrix elements (via the chain rule through LoRA factors and omega coefficients) or with respect to the LoRA factors themselves, nor does it specify how the omega coefficients enter the importance computation. Without this clarification, the selection of the top-p matrices is not reproducible, and the comparison with the Random baseline in Table 4 does not establish that the importance estimates are meaningful.","section":"Section 3.2, Eqs. (6)-(10)"},{"comment":"The top-p percentage is set to 5% or 10% empirically, but the manuscript does not report which value is used for which dataset and backbone, nor does it provide a sensitivity analysis over p. Since p directly controls the number of frozen matrices and trades off plasticity against stability, the absence of this information makes it difficult to judge how robust the reported gains are to this hyperparameter. Please report the p values used in all experiments and add a sensitivity study (e.g., varying p as in the rank and lambda studies).","section":"Section 5.1.3 and Appendix A"},{"comment":"The task-ID inference uses the feature extractor f(x, Theta_1), but Theta_1 is not defined as a stored snapshot. After task 1, IPC freezes only selected matrices, and later tasks update both the omega coefficients and the remaining matrices, so the current model after all tasks is not equal to Theta_1. Please clarify whether a copy of the model immediately after task 1 is saved and used for prototype and covariance estimation, or whether the current composed model is used. This matters because the reported accuracy with task-ID inference (e.g., 92.86 on Split CIFAR-100 in Table 1) depends on the correctness of the task-ID inference.","section":"Section 4.1, Eq. (13)"}],"minor_comments":[{"comment":"The stated ranges 'beta1 > 0, beta2 < 1' are imprecise; for the exponential moving average to be stable, one normally requires 0 < beta1 < 1 and 0 < beta2 < 1. The experiments use 0.85 for both, so please correct the conditions.","section":"Section 3.2, Eq. (7)"},{"comment":"The caption of Figure 1(b) describes the right columns as 'variation in important parameters', but the text defines importance via the sensitivity score while the variation is an L2 norm of parameter change. Please clarify the relationship between the two quantities, or revise the figure caption to match the text.","section":"Figure 1 and Section 3.2"},{"comment":"The header 'PTM Mehods' contains a typo, and the entries for Split DomainNet are missing for the Sup-21K backbone, which makes the reproducibility of the experimental setup incomplete. Please correct the header and report the weight types and ranks for all datasets used with each backbone.","section":"Table A.7"},{"comment":"The footnote markers '†' and '‡' appear next to several method names, but the caption does not unambiguously indicate which rows each marker applies to. Please make the footnotes explicit.","section":"Table 1 caption"},{"comment":"The qualitative claim that 'with IPC, the positions of important parameters do not overlap largely' is supported only by visual inspection of the yellow boxes. A quantitative overlap measure (e.g., Jaccard index between the sets of top-p matrices across tasks) would strengthen this claim.","section":"Section 5.3, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central ambiguity in Section 3.3 is whether the released code actually implements the freezing as described. I did not run the code; if the code contains masking or snapshotting that resolves the ambiguity, the paper should be revised to state the mechanism explicitly and align Eq. (1) and Eq. (11) with the implementation. The empirical results are extensive and the multimodal evaluation is a strength, but the manuscript in its current form does not unambiguously support the claimed mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is empirically solid and the combination of orthogonal LoRA composition with importance-based freezing is a reasonable next step in PTM-based continual learning. But the central IPC mechanism is under-specified in a way that matters. Section 3.3 says you update the omega coefficients of all previous LoRA modules with a low learning rate; Eq. (1) shows every parameter matrix is a sum over those modules. That means a matrix you 'freeze' still changes whenever omega moves, unless you also mask omega updates for frozen matrices. The objective in Eq. (11) doesn't show any mask. So either the freeze is not a freeze as described, or the paper's equations miss the masking. The headline gains on Split CIFAR-100 are attributed to IPC, so this needs to be pinned down before the causal story holds. The code may resolve it, but a reader shouldn't have to infer from the repo what the paper claims.\n\nCredit where due: the benchmark coverage is genuinely broad—four standard CL splits plus two multimodal datasets—and the ablations are more thorough than typical for this area. Rank and lambda sweeps are there; IPC is compared to random selection; the incremental component analysis in Table 5 is helpful. The QR-based orthogonal composition is a clean and easy-to-reproduce extension of the LoRA-FA/InfLoRA ideas. The observation that important parameters drift under orthogonal LoRA is interesting and does motivate the freezing idea, though the evidence in Fig. 1 is visual, not quantitative.\n\nThe soft spots are mostly about rigor of presentation. The top-p fraction (5% vs 10%) is set per dataset without a sensitivity curve. The drift claim would be stronger with a number, e.g., how many of the top-p matrices move by more than some threshold. The multimodal experiments use two datasets both from the same group as the authors; that's fine as evidence of generality but doesn't independently validate.\n\nMy take: if the authors clarify the omega/freeze interaction—say whether omega updates are masked for frozen matrices or whether only the new LoRA module is blocked—the method likely works as claimed. As written, the ambiguity is major enough that the method section needs revision, not just polish. This deserves a serious referee; the experiments and the empirical phenomenon make it worth the time. But I wouldn't cite the mechanism until the update rule is stated unambiguously.","headline":"Solid experiments and a plausible combination of known LoRA ideas, but the freezing mechanism is under-specified: 'frozen' matrices may still change through global omega updates, so the central IPC claim needs a clearer update rule.","tokens_in":20058,"tokens_out":4021,"would_cite":false,"duration_ms":34968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that under orthogonal LoRA tuning the parameters most critical to previous tasks still drift, and that freezing the top-p important parameter matrices—via LoRAC-IPC, an orthogonal LoRA composition with importance-based…","keywords":["continual learning","class-incremental learning","catastrophic forgetting","LoRA","low-rank adaptation","orthogonal LoRA composition","parameter importance","rehearsal-free learning"],"falsifier":"Track every matrix frozen after task $t$ through all later tasks: if any of its entries changes by more than numerical precision after training task $t+1$, the freeze is not being enforced and the claimed mechanism is not what produces the gains. A simpler check is to hold $\\omega$ fixed at 1 for all past LoRA modules; if accuracy and forgetting barely change, the low-rate $\\omega$ updates claimed to aid plasticity play no measurable role.","tokens_in":19027,"feed_emoji":"❄️","tokens_out":9691,"duration_ms":77022,"temperature":0.7,"pith_summary":"Under orthogonal LoRA tuning, the parameters most critical to earlier tasks still change noticeably once later tasks are learned, so orthogonal constraints alone do not stop forgetting. The paper proposes LoRAC-IPC: compose sequentially learned LoRA modules on a pre-trained Vision Transformer, enforce orthogonality among their subspaces through QR decomposition and a regularization loss, and after each task freeze the top-p parameter matrices ranked by a smoothed sensitivity-uncertainty importance score. On several rehearsal-free continual learning benchmarks the method reports the best or near-best accuracy, with headline gains of 6.35% higher accuracy and 3.24% lower forgetting on Split CIFAR-100 with a Sup-21K backbone. If correct, this gives a simple memory-free recipe for keeping pre-trained model fine-tuning stable across long task sequences.","feed_headline":"Freezing critical parameters lifts continual learning by 6.35%","feed_subtitle":"A rehearsal-free LoRA method freezes the most important ViT parameter matrices before each new task.","key_machinery":"The load-bearing identity is the LoRA composition $W_t = W_0 + \\omega_1\\Delta W_1 + \\cdots + \\omega_t\\Delta W_t$, with each $\\Delta W_\\tau$ reparameterized through QR decomposition as $\\Delta W_\\tau = Q_\\tau K_\\tau$, where the columns of $Q_\\tau$ are orthonormal. A regularization loss $\\mathcal{L}_{\\mathrm{ortho}}(\\tilde{Q}_t) = \\|\\tilde{Q}_t^\\top \\tilde{Q}_t - I\\|^2$ on the concatenated matrix $\\tilde{Q}_t = [Q_1,\\ldots,Q_t]$ forces subspaces learned for different tasks to be mutually orthogonal. The IPC component then ranks each parameter matrix by its average importance score $\\bar{I}\\cdot\\bar{U}$ and freezes the top-p matrices before the next task, holding fixed the parameters that mattered most for the current task. The $\\omega$ coefficients are updated with a low learning rate to preserve some plasticity, which slightly relaxes the orthogonality constraint.","core_discovery":"The paper's central discovery is that LoRA-level orthogonality does not protect the parameters that matter: even when the columns of the learned LoRA factors are mutually orthonormal, the parameter matrices most sensitive to the current task's loss still change substantially after later tasks are learned. The authors attribute this to the low-rank nature of LoRA—the low-rank subspace cannot represent the original parameter space, so orthogonal low-rank solutions do not imply orthogonal full matrices. Their fix is to freeze, before each new task, the top-p parameter matrices selected by the importance score $S(w_{t,ij}) = \\bar{I}(w_{t,ij}) \\cdot \\bar{U}(w_{t,ij})$, a product of exponentially smoothed gradient sensitivity and smoothed uncertainty, while composing past LoRA modules with learnable weights and QR-decomposed orthonormal factors. The reported outcome is rehearsal-free continual learning that surpasses previous methods on Split CIFAR-100, Split ImageNet-R, 5-datasets, and Split DomainNet, with the largest gains on Split CIFAR-100 (6.35% higher accuracy, 3.24% lower forgetting under Sup-21K).","pith_inferences":["The paper tests the freeze-first-then-compose recipe only with LoRA; the same importance-selection and freezing logic could plausibly be applied to adapters or prompts composed orthogonally, a transfer the paper does not claim.","Because the importance score is computed on mini-batches of the current task, the top-p selection may be noisy early in training; using importance estimated after convergence or with validation data is a natural robustness check the paper does not perform.","The method relies on task-ID inference at test time for its best numbers; in boundary-free or blurry continual learning where task boundaries are unknown, the gains would likely shrink unless the inference step is replaced."],"forward_implications":["On Split CIFAR-100 with the Sup-21K backbone, LoRAC-IPC reports 6.35% higher average accuracy and 3.24% lower forgetting than previous rehearsal-free methods.","On Split ImageNet-R it reports 1.24% higher accuracy than a strong projection-based baseline, and on the self-supervised MoCo-1K backbone it beats a full fine-tuning method by 2.39%.","On the 5-datasets benchmark it reaches 95.58% average accuracy with 0.03% forgetting (95.77% and 0.01% with batch-wise task-ID inference), close to the joint-training upper bound.","The ablation attributes 1.96% accuracy to LoRA composition, another 4.43% to the orthogonality loss, 0.45% to the important-parameter constraints, and 0.89% to task-ID inference on Split CIFAR-100."],"supporting_citations":[{"why":"supplies the low-rank update decomposition $\\Delta W = AB$ that every LoRA module in the composition relies on.","marker":"[23]"},{"why":"introduces the QR decomposition of the projection-down matrix $A$, reused here to obtain orthonormal columns $Q_t$ for the orthogonal composition.","marker":"[31]"},{"why":"demonstrates orthogonal LoRA tuning for continual learning in language models, the starting point this paper extends and critiques.","marker":"[26]"},{"why":"proposes interference-free LoRA with orthogonality constraints, the main prior approach whose assumption this paper tests and finds insufficient.","marker":"[27]"},{"why":"states the orthogonal-gradient principle that motivates why orthogonal updates should in principle leave old-task loss unchanged.","marker":"[28]"},{"why":"supplies the smoothed sensitivity-uncertainty importance score used by the IPC component to rank parameter matrices.","marker":"[43]"},{"why":"provides the Mahalanobis nearest-class-mean task-ID inference used at test time and serves as a strong rehearsal-free baseline.","marker":"[42]"},{"why":"defines the rehearsal-free prompt-based continual learning setting and serves as a baseline on the benchmarks.","marker":"[16]"},{"why":"provides another prompt-based rehearsal-free baseline that LoRAC-IPC is compared against for the headline gains.","marker":"[17]"}],"fun_headline_variants":["Freeze critical LoRA params for 6.35% better continual learning","LoRAC freezes top ViT matrices, gains 6.35% in continual learning","Critical parameter freezing outperforms orthogonal LoRA on CL","Freezing key LoRA matrices reduces forgetting by 3.24%","Orthogonal LoRA not enough: freeze critical params for CL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on trusting the sensitivity-uncertainty score to pick the truly critical parameter matrices, and on assuming that freezing those matrices is not silently undone by the later low-learning-rate updates to the weighting coefficients $\\omega$.","fun_headline_variants_meta":{"raw":{"variants":["Freeze critical LoRA params for 6.35% better continual learning","LoRAC freezes top ViT matrices, gains 6.35% in continual learning","Critical parameter freezing outperforms orthogonal LoRA on CL","Freezing key LoRA matrices reduces forgetting by 3.24%","Orthogonal LoRA not enough: freeze critical params for CL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00091,"raw_usage":{"total_tokens":3929,"prompt_tokens":985,"completion_tokens":2944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2848}},"tokens_in":601,"tokens_out":2944,"duration_ms":19995,"temperature":1.0,"reasoning_tokens":2848,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:09:10.763181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track every matrix frozen after task $t$ through all later tasks: if any of its entries changes by more than numerical precision after training task $t+1$, the freeze is not being enforced and the claimed mechanism is not what produces the gains. A simpler check is to hold $\\omega$ fixed at 1 for all past LoRA modules; if accuracy and forgetting barely change, the low-rate $\\omega$ updates claimed to aid plasticity play no measurable role.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the low-rank update decomposition $\\Delta W = AB$ that every LoRA module in the composition relies on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"demonstrates orthogonal LoRA tuning for continual learning in language models, the starting point this paper extends and critiques."},{"cited_title":"Liang, W.-J","cited_arxiv_id":null,"evidence_quote":"proposes interference-free LoRA with orthogonality constraints, the main prior approach whose assumption this paper tests and finds insufficient."},{"cited_title":"Farajtabar, N","cited_arxiv_id":null,"evidence_quote":"states the orthogonal-gradient principle that motivates why orthogonal updates should in principle leave old-task loss unchanged."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"supplies the smoothed sensitivity-uncertainty importance score used by the IPC component to rank parameter matrices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the rehearsal-free prompt-based continual learning setting and serves as a baseline on the benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides another prompt-based rehearsal-free baseline that LoRAC-IPC is compared against for the headline gains."}],"review_version":1}