{"id":"04158875-aef8-4743-92ea-52d5418b228a","arxiv_id":"2501.05017","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CKPD-FSCIL uses covariance-guided weight decomposition and adaptive layer selection to learn new classes incrementally without changing the model's architecture or inference cost.","lead":"This paper presents a method for few-shot class-incremental learning that decomposes pretrained neural network weights into frozen, knowledge-protecting parts and learnable, redundant parts, then adaptively selects which layers to update. The method claims state-of-the-art accuracy with no added parameters or compute at inference time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1 motivates decomposing WΣ_in^{1/2}, not WΣ_in; the paper's SVD ranking and ASR can therefore select the wrong frozen/redundant subspaces.","rationale":"The paper has real strengths: the residual definition in Eq. 10 guarantees exact output preservation at decomposition time; the merge in Eq. 11 delivers zero inference overhead; the five-seed stability table (Tab. 10) and training-time accounting (Tab. 12) are useful; and the layer-selection pattern in Fig. 5 is an interesting empirical finding. The reader is right, however, that the paper's mechanism is not justified. I want to stress that the concern is not merely 'the theorem is unproven' in an abstract sense: Eq. 1 points to WΣ_in^{1/2}, while Eq. 4 decomposes WΣ_in; these differ whenever Σ_in is not a scalar multiple of the identity. In a ViT, feature covariances are strongly anisotropic, so the difference is material. The same miscalibration infects the ASR score, meaning even if the reported numbers are reproducible, they would not validate the advertised 'knowledge-preserving' decomposition. Given that the main tables also contain multiple unnamed CKPD variants (Tables 2-4) and the miniImageNet ViT-B-IN21K row is not above PriViLege in average accuracy, the central claim as stated remains unsupported. My concern reinforces the reader's REJECT verdict; I do not see a reason to change it, but the proposed square-root decomposition experiment could turn this concern into a constructive fix.","tokens_in":29231,"tokens_out":10157,"duration_ms":103273,"concrete_test":"Use the public ViT-B/16-CLIP backbone and CIFAR-100 base session. For each of the 48 linear layers, compute (i) the top-r subspace from SVD of WΣ_in as in Eq. 4, (ii) the top-r subspace from SVD of WΣ_in^{1/2}, and (iii) the top-r subspace of the between-class scatter matrix B = Σ_c (μ_c−μ)(μ_c−μ)^T using the same covariance sample set. Report the average principal-angle distances among (i), (ii), and (iii). If (i) is no closer to (ii) or (iii) than to a random subspace, the decomposition is selecting directions by input variance rather than output discriminative structure. Then run the full CKPD pipeline with M=WΣ_in^{1/2} replacing M=WΣ_in (same rank r, same K, same training protocol) and compare base-class retention, average accuracy, and PD. If the alternative preserves old classes better or achieves higher average accuracy, the paper's core assumption is empirically falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that SVD of WΣ_in (Eq. 4) ranks directions of W by their contribution to discriminative output structure. The paper justifies this with Σ_out = WΣ_in W^T (Eq. 1), but that identity implies no such ranking. For M = WΣ_in with SVD M = U S V^T, one has W = U S V^T Σ_in^{-1} and therefore Σ_out = WΣ_in W^T = U S (V^T Σ_in^{-1} V) S^T U^T, which is not diagonal in general. The matrix whose SVD genuinely diagonalizes the output covariance is WΣ_in^{1/2}; decomposing WΣ_in instead ranks by WΣ_in^2 W^T, so the top singular directions can be dominated by high input variance even when they carry no class-discriminative information. Eq. 3 compounds this by using a raw second moment (one sample per class, not centered, not a between-class scatter), which is not the LDA-style structure invoked in the motivation. The flaw propagates to Eq. 6 and to ASR (Eq. 13). Moreover, a 'small' s_i does not imply a small effect in the original input space: the corresponding component is s_i u_i (Σ_in^{-1} v_i)^T, whose operator norm can be large when Σ_in^{-1} stretches v_i. Thus the bottom subspace is not guaranteed safe, and the top subspace is not guaranteed knowledge-critical. The exact-residual construction W_frozen = W − BA (Eq. 10) only guarantees no initial shift; it does not fix the ranking. If the ranking is wrong, both stability and the layer-selection signal are compromised, and the central claim is not supported by its mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CKPD-FSCIL, a few-shot class-incremental learning method that decomposes selected linear layers of a ViT backbone into a frozen 'knowledge-sensitive' subspace and a learnable low-rank 'redundant' subspace, then merges the learned adapters back into the weights after each session. Layer selection is driven by an Adapter Sensitivity Ratio (ASR) computed from the same decomposition. The method is evaluated on CUB-200, CIFAR-100, miniImageNet, ImageNet-1K, and a standard CIL benchmark, reporting state-of-the-art average accuracies and low performance drops while adding zero parameters and zero FLOPs at inference.","tokens_in":29401,"tokens_out":3725,"duration_ms":37077,"significance":"If the mechanism worked as claimed, the paper would make a useful contribution: it targets two under-explored aspects of FSCIL (intra-layer redundancy and layer-wise sensitivity), provides a continuous recalibration scheme, and achieves zero inference overhead by weight merging. The manuscript has concrete strengths: the code is released, the experiments cover several backbones and benchmarks, the appendix-style ablations include adapter dropout and random-seed robustness, and the efficiency analysis separately measures offline and online training cost. However, the central theoretical motivation for the decomposition is mathematically flawed, and the reported tables contain unexplained duplicate rows. The contribution is therefore not currently established.","major_comments":[{"comment":"The SVD of WΣ_in does not diagonalize the output covariance that the motivation is built on. From Eq. (1), if WΣ_in = U S V^T then W = U S V^T Σ_in^{-1}, so Σ_out = WΣ_in W^T = U S (V^T Σ_in^{-1} V) S^T U^T, which is not diagonal in general because V^T Σ_in^{-1} V is not diagonal. The matrix whose SVD genuinely diagonalizes Σ_out is WΣ_in^{1/2}, not WΣ_in. Thus the singular vectors and singular values of WΣ_in do not, by the paper's own definition, isolate the directions of W that are most important for the model's discriminative output structure.","section":"Sec. 3.2.1, Eq. (1)-(4)"},{"comment":"The 'input covariance' used in the method is a raw second moment E[xx^T] estimated from one sample per class, not a centered covariance and not a between-class scatter. The theoretical motivation appeals to classical discriminant analysis, whose relevant object is inter-class versus intra-class scatter; Eq. (3) cannot capture intra-class covariance with one sample per class. The claim that the resulting spectrum ranks knowledge-sensitive directions is therefore unsupported by the stated LDA-style motivation.","section":"Sec. 3.2.1/3.2.2, Eq. (3)"},{"comment":"A small singular value s_i of WΣ_in does not imply that the corresponding component has a small effect in the original input space. After the reconstruction in Eq. (5), the i-th component is s_i u_i (Σ_in^{-1} v_i)^T, whose operator norm is s_i times the norm of Σ_in^{-1} v_i; the inverse covariance can amplify a small s_i considerably. Consequently, the bottom-r subspace is not guaranteed to be safe to adapt, and the top subspace is not guaranteed to be knowledge-critical. The exact-residual construction of W_frozen in Eq. (10) only guarantees that the initial output is unchanged; it does not repair the incorrect ranking.","section":"Sec. 3.2.2, Eq. (5)-(10)"},{"comment":"The ASR metric is defined from the same singular value decomposition used to create the redundant subspace: a low ASR is, by construction, the condition that the bottom-r singular values have small total energy. The sentence 'A low ASR indicates that the redundant subspace contributes minimally to the layer's overall function' is therefore definitionally aligned with the decomposition rather than an independent measure of forgetting risk. The empirical comparison against manual and uniform layer selection in Fig. 4 provides useful evidence, but it does not validate ASR as a measure of safety outside the paper's own decomposition choice.","section":"Sec. 3.3.1, Eq. (13)"},{"comment":"The main comparison tables contain multiple CKPD-FSCIL rows with the same backbone but different base session accuracies and no identifying configuration: Table 2 has two ViT-B-CLIP rows, Table 3 has three ViT-B-CLIP rows, and Table 4 has two ViT-B-CLIP rows. Without labels explaining differences in training protocol, hyperparameters, or settings, the claim that CKPD-FSCIL 'consistently outperforms' state-of-the-art approaches is ambiguous and the results are not reproducible from the paper as written.","section":"Tables 2, 3, 4"}],"minor_comments":[{"comment":"The sentence 'While effective, these methods face several critical:' is incomplete and should be reworded.","section":"Sec. 2.2"},{"comment":"There is a duplicated word in 'KANet [22] and and CA-CLIP [20]' that should be corrected.","section":"Sec. 2.2"},{"comment":"The text says features are centered by Layer Normalization, but Eq. (3) is a raw second moment without subtracting a mean; the notation and the prose should be made consistent.","section":"Sec. 3.2.2, Eq. (3)"},{"comment":"The figure axis labels use 'ffn0, ffn1' while the text and Fig. 2 refer to 'FFN1, FFN2'; the naming should be unified.","section":"Fig. 5"},{"comment":"Table 12 states that CUB-200 incremental sessions are trained for 2000 iterations, while Sec. 4.1 specifies 1000 iterations for CUB-200; the inconsistency should be resolved.","section":"Table 12"},{"comment":"The paragraph 'At the start of each incremental session, ALS computes...' contains a repeated sentence; one of the two occurrences should be removed.","section":"Sec. 3.3.2"}],"recommendation":"reject","confidential_remarks":"The empirical results are extensive and the zero-overhead design is attractive, but the mathematical core of the method is not sound as written: the SVD of WΣ_in does not diagonalize the output covariance, the covariance estimate is not a centered or class-discriminative statistic, and the reconstruction with Σ_in^{-1} can amplify small singular components. These are load-bearing errors that affect the decomposition, the safety guarantee, and the ASR-based layer selection. A correction would require changing the decomposition to WΣ_in^{1/2} (or another principled object) and re-running all experiments, which is beyond a minor revision. The unexplained duplicate rows in the main tables also need an audit if the work is resubmitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you work on FSCIL. The core idea—find redundant subspaces inside pretrained weights via covariance-guided SVD and adapt only those, reselecting layers each session—is genuinely worth discussing. The CALS/ASR layer selection has a striking pattern: it avoids middle blocks and favors early/late ones, and that is a testable inductive bias. Zero inference overhead after merging is a real advantage over prompt/adapter methods that pile up modules. Credit where due: the framework is new in this combination, they ship code, and the efficiency numbers are clean.\n\nBut the central claim \"consistently outperforms SOTA\" is not supported by the paper's own tables. On miniImageNet ViT-B-IN21K, CKPD-FSCIL is slightly below PriViLege (95.22 vs 95.27 AVG). On CIFAR-100, margins over PriViLege are under half a percent with no error bars. Worse, the main tables list multiple CKPD-FSCIL rows with different base accuracies for the same backbone/dataset, and Table 7 shows CKPD at 71.3% AVG on CIFAR-100 while main tables report 84.66–88.36. The official configuration is ambiguous and the ablation conflicts with the main results.\n\nThe theoretical motivation has a real flaw. They define knowledge via output covariance Σ_out = WΣ_in W^T and claim SVD of WΣ_in ranks knowledge-sensitive directions. The identity does not imply that: the SVD that diagonalizes the output covariance is WΣ_in^{1/2}, not WΣ_in. Decomposing WΣ_in ranks by WΣ_in^2 W^T, which can be dominated by high input variance directions that are not class-discriminative. Also, a small singular value does not mean small effect in the original space: the component is s_i u_i (Σ_in^{-1} v_i)^T, whose norm can be large. So the frozen subspace is not provably knowledge-critical and the redundant subspace is not provably safe. The exact-residual construction only guarantees no initial shift. The ASR then inherits this issue.\n\nThe empirical layer selection results are still useful, but the paper needs a rigorous justification or a clear empirical validation that the selected subspaces actually align with knowledge—for example, via the adapter-dropout or forgetting analysis redone per subspace direction. Also: one canonical configuration, error bars, and a reconciled ablation are mandatory.\n\nWould I referee it? Yes, even though I'd send it back for major revision. The idea has legs and the layer-selection observation is valuable. But as submitted, the central claim is overreaching.","headline":"A conceptually appealing but currently overclaimed FSCIL method: the adaptive layer selection is the genuinely useful part, but the theoretical motivation does not hold up and the evidence tables do not support consistent SOTA.","tokens_in":30121,"tokens_out":2314,"would_cite":false,"duration_ms":20512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims covariance-guided decomposition of pretrained weights lets few-shot class-incremental learning beat existing methods with zero inference overhead.","keywords":["few-shot class-incremental learning","catastrophic forgetting","knowledge-preserving decomposition","adaptive layer selection","low-rank adaptation","input covariance","vision transformer","stability-plasticity balance"],"falsifier":"Compute the same CKPD pipeline on CIFAR-100 but replace $\\Sigma_{\\mathrm{in}}$ with a random matrix that has the same singular values and is rotationally uncorrelated with the data; if the resulting incremental accuracy matches the true-covariance version, then the covariance guidance is not doing the work the paper claims. A sharper check: after the frozen subspace is built, zero out or randomly perturb $W_{\\mathrm{frozen}}$ and measure base-class accuracy; if it drops as much as perturbing $W_{\\mathrm{learnable}}$, the top singular directions are not the knowledge-bearing ones.","tokens_in":28849,"feed_emoji":"🧠","tokens_out":6098,"duration_ms":57180,"temperature":0.7,"pith_summary":"The paper tries to establish that a pretrained vision transformer already contains enough unused capacity to keep learning new classes without forgetting, provided each linear layer is split into a frozen knowledge-sensitive part and a learnable redundant part. It claims the correct split can be found by multiplying the weight matrix $W$ by the input covariance $\\Sigma_{\\mathrm{in}}$ and taking a singular value decomposition: the top singular directions carry prior knowledge, the bottom $r$ form a safe low-rank adapter. A second mechanism ranks layers by the fraction of energy in that redundant subspace (the Adapter Sensitivity Ratio) and adapts only the safest layers. If right, the method achieves top accuracy on several benchmarks while adding zero parameters and zero extra FLOPs at inference, because each session's adapter is merged back into the original weights.","feed_headline":"Adapting only redundant weight subspaces sets new FSCIL records","feed_subtitle":"The method keeps old classes by freezing knowledge-bearing directions and merges learnable adapters back after each session.","key_machinery":"The load-bearing object is the covariance-weighted singular value decomposition of $W\\Sigma_{\\mathrm{in}}$ with reconstruction through $W=USV^{\\top}\\Sigma_{\\mathrm{in}}^{-1}$. Its singular values rank each input-to-output direction by how strongly it shapes the output covariance, which the paper identifies with task-discriminative knowledge. The bottom $r$ singular components are factorized into a low-rank adapter $BA$; the top $R-r$ define $W_{\\mathrm{frozen}}$ as the residual $W-BA$, guaranteeing zero initial drift. The Adapter Sensitivity Ratio, $\\mathrm{ASR}(\\ell)=\\sum_{i=R-r+1}^{R}s_i/\\sum_{i=1}^{R}s_i$, then selects the $K$ layers whose redundant subspace carries the smallest share of energy, because changing those layers is safest.","core_discovery":"The central discovery is that covariance-guided decomposition of pretrained weights turns the stability-plasticity trade-off into a subspace-allocation problem. For a layer with weight $W$ and input covariance $\\Sigma_{\\mathrm{in}}$, the output covariance satisfies $\\Sigma_{\\mathrm{out}}=W\\Sigma_{\\mathrm{in}}W^{\\top}$, so the directions of $W\\Sigma_{\\mathrm{in}}$ that most amplify input structure are the ones that matter for previously learned classes. The paper defines the frozen subspace from the top $R-r$ singular components and the learnable adapter from the bottom $r$, reconstructing $W$ exactly as $W_{\\mathrm{frozen}}+BA$ so that no prediction changes before training. The decomposition is recomputed each session with updated covariance statistics, and the layer selection is redone with the Adapter Sensitivity Ratio, the share of a layer's total singular-value energy in its redundant subspace. The claim is that this yields both higher adaptability and better retention than freezing, full fine-tuning, or adding external prompt and adapter modules.","pith_inferences":["The same machinery should transfer to other transformer-based continual settings, since it only requires linear layers, an input covariance estimate, and a mergeable low-rank update; language models and multimodal models are natural candidates.","An implicit consequence is that the amount of safe adaptation capacity is bounded by how much truly redundant spectral energy the pretrained weights contain; on heavily compressed or highly optimized models, the bottom-$r$ subspace may be too small to carry new classes, and accuracy gains would shrink.","One testable extension is to replace the single random sample per class in the covariance buffer with a small ensemble of samples: the paper reports low seed variance, but measuring whether covariance quality degrades as class count grows would indicate when the one-sample buffer stops being sufficient.","The ASR ranking could itself be used as a no-validation heuristic for choosing adapter rank per layer, since the metric already reports how much energy would be touched by adaptation."],"forward_implications":["If the claim holds, FSCIL no longer requires a choice between freezing the backbone and accumulating per-session modules: the same parameter count and inference cost suffice.","Because adapters are merged after each session, deployed models can keep growing in knowledge without growing in size, which matters for on-device continual learning.","The continuous recalibration of the decomposition predicts that static decompositions will drift out of alignment, and the paper reports that the continuous version outperforms the one-shot version on CIFAR-100 and CUB-200.","The ASR-based selection implies that layer choice can be automated from data statistics instead of manual heuristics, and the paper's selection maps largely avoid middle transformer blocks.","Applying the same low-rank repurposing inside an existing continual-learning method improves its accuracy, suggesting the mechanism transfers beyond few-shot settings."],"supporting_citations":[{"why":"Establishes the FSCIL problem definition and the benchmark protocol that the paper's experiments follow.","marker":"[1]"},{"why":"Provides the covariance-oriented decomposition idea that the paper adapts and extends to continual learning.","marker":"[30]"},{"why":"Introduces mergeable low-rank adaptation, the adapter form the paper uses and a key baseline it compares against.","marker":"[63]"},{"why":"Introduces SVD-based principal-subspace initialization, compared in the ablation of decomposition strategies.","marker":"[64]"},{"why":"Introduces activation-weighted singular value decomposition, the direct predecessor of the covariance-weighted approach.","marker":"[65]"},{"why":"Provides the DR loss, the ETF classifier, and a strong FSCIL baseline whose training choices the paper adopts.","marker":"[16]"},{"why":"Provides the SSM projector and a strong FSCIL baseline whose setup the paper follows.","marker":"[17]"}],"fun_headline_variants":["Covariance-guided decomposition reuses redundant subspace for FSCIL","Zero-overhead adapter merged after each session boosts FSCIL","Adaptive layer selection via sensitivity ratio improves FSCIL","Reuse redundant weight directions to beat FSCIL stability-plasticity","Frozen subspace for old, learnable for new: CKPD-FSCIL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the assumption that the directions of the weight matrix that most amplify the input-data covariance are exactly the directions needed to remember old classes, so cutting off the smallest directions leaves knowledge intact.","fun_headline_variants_meta":{"raw":{"variants":["Covariance-guided decomposition reuses redundant subspace for FSCIL","Zero-overhead adapter merged after each session boosts FSCIL","Adaptive layer selection via sensitivity ratio improves FSCIL","Reuse redundant weight directions to beat FSCIL stability-plasticity","Frozen subspace for old, learnable for new: CKPD-FSCIL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3677,"prompt_tokens":1044,"completion_tokens":2633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2541}},"tokens_in":660,"tokens_out":2633,"duration_ms":18778,"temperature":1.0,"reasoning_tokens":2541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:36.016121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same CKPD pipeline on CIFAR-100 but replace $\\Sigma_{\\mathrm{in}}$ with a random matrix that has the same singular values and is rotationally uncorrelated with the data; if the resulting incremental accuracy matches the true-covariance version, then the covariance guidance is not doing the work the paper claims. A sharper check: after the frozen subspace is built, zero out or randomly perturb $W_{\\mathrm{frozen}}$ and measure base-class accuracy; if it drops as much as perturbing $W_{\\mathrm{learnable}}$, the top singular directions are not the knowledge-bearing ones.","supporting_citations":[{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Introduces mergeable low-rank adaptation, the adapter form the paper uses and a key baseline it compares against."},{"cited_title":"Neural collapse inspired feature-classifier alignment for few-shot class-incremental learn- ing,","cited_arxiv_id":null,"evidence_quote":"Provides the DR loss, the ETF classifier, and a strong FSCIL baseline whose training choices the paper adopts."}],"review_version":1}