{"id":"2a83a5d8-5dc3-4895-b342-460093613359","arxiv_id":"2509.21606","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FedProTIP combines gradient projection onto orthogonal complements of prior-task subspaces with subspace-relevance task-identity prediction to achieve replay-free, task-agnostic federated continual learning.","lead":"This paper introduces FedProTIP, a federated learning method that lets a shared model learn new tasks in sequence without storing old data, by steering each update away from previously learned feature directions. It also infers which task a test sample comes from, removing the usual assumption that task identity is known at inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4 projects client gradients on the wrong side of the weight matrix: as written, updates do not satisfy the invariance in Eq. 3, so the central forgetting-mitigation mechanism is mathematically unsupported.","rationale":"The reader's weakest assumption concerned the representativeness of random activation sampling (m_s << m). That is a legitimate scalability concern, but it presupposes the projection mechanism works when the subspace is correctly estimated. The more fundamental issue is that Eq. 4, as written, does not implement the standard GPM projection. GPM's invariance guarantee requires the weight update to satisfy ∆W_l Φ = 0; this is obtained by right-multiplying the gradient by (I - ΦΦ^T). Eq. 4 left-multiplies. For d_out ≠ d_in the left-multiplied projection acts on the output dimension and generally leaves ∆W_l Φ nonzero, so the step in Eq. 3 approximating the preserved activations is unjustified. This is not a stylistic issue: if the code follows the equations, the method should forget previous tasks; if the code deviates, the paper misdescribes its own algorithm. The reader's point about Eq. 9 normalization is related but secondary: even after fixing the multiplication side, the aggregated bases must be re-orthonormalized for ΦΦ^T to be a projector. The experiments are extensive and the gains are large, so I am not claiming the empirical result is fraudulent; the correct resolution is to check the implementation and correct the equations. Hence CONDITIONAL, with the condition being verification of the projection side and normalization, rather than a demand for new benchmarks.","tokens_in":22971,"tokens_out":10832,"duration_ms":99994,"concrete_test":"Inspect the authors' released code for the projected-gradient step. If the update is implemented as `grad = grad - Phi @ (Phi.T @ grad)`, run the minimal two-task test: let d_in=2, d_out=2, task A inputs all equal (1,0); extract Phi from task A activations; train task B with this update; evaluate the final model on task A inputs. If outputs change materially, Eq. 3 is violated and the left-multiplied projection is not the forgetting-prevention mechanism. If the code instead uses `grad = grad - (grad @ Phi) @ Phi.T` (or equivalently right-multiplies by I - Phi@Phi.T), then the empirical results may stand and the paper needs only a correction of Eqs. 4/14 and a normalization note for Eq. 9; without code access, ask the authors to state which side of the weight matrix the projector is applied to and to verify Eq. 3 numerically.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section 3.2 the core bases Φ_l^(t) are extracted from input activations a_l^(t) ∈ R^{d_l×m}, so Φ_l spans the input space of layer l. For Eq. 3 to hold, subsequent updates must leave prior activations fixed: for every previous feature vector a, ∆W_l a = 0, equivalently ∆W_l Φ_l^(1:t) = 0. That condition is enforced by right-multiplying the gradient with (I - ΦΦ^T). Eq. 4 instead computes ∆W_k ← (I - ΦΦ^T) ∇W_k, a left multiplication. With a rectangular weight matrix this does not imply ∆W_k Φ = 0; in fact (I - ΦΦ^T)∇W_k Φ can be nonzero, so the change in the pre-activation for a previous-task input is not zero and Eq. 3 does not follow. Thus, taken literally, the written FedProTIP update does not implement the orthogonal-projection mechanism claimed to mitigate forgetting. The reported near-zero forgetting would then have to come from the task-identity routing, from an unstated right-multiplication correction in the code, or from empirical confounds. Appendix B repeats the same left-multiplied P = I - ΦΦ^T and asserts P^2 = P, which also requires Φ to have orthonormal columns; Eq. 9 appends unnormalized residuals, so even the stated projector is not idempotent as written. This is the most load-bearing weakness because it attacks the theoretical core, not just the representativeness of the sampled activations.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FedProTIP is a replay-free federated continual learning framework with two components. First, to mitigate catastrophic forgetting, each client projects its local gradients onto the orthogonal complement of subspaces spanned by the activated representations of previously learned tasks; per-client core bases are extracted by randomized SVD on a small random subset of layer-wise activations (Section 4.2) and aggregated on the server into a global task subspace (Section 4.3). Second, to enable task-agnostic inference, each client stores reference vectors that measure how its own final-layer activations align with each task subspace; at test time, a relevance vector computed from the test activation is compared with the stored references via cosine similarity, and a majority vote across clients predicts the task identity, which routes the input to the corresponding output head (Section 4.4). Experiments on CIFAR100, ImageNet-R (5/10/20 splits), and DomainNet report consistent gains over FedAvg, GLFC, LGA, TARGET, LANDER, and FOT, with particularly large margins in task-agnostic class-incremental settings (e.g., 86.00 vs 41.80 average accuracy on 10-split CIFAR100, Table 13), together with lower training time, GPU memory, and communication cost. The appendix reports ablations over client counts, projection thresholds, batch sizes, backbones trained from scratch or pretrained, task orderings, and task-prediction accuracy per phase.","tokens_in":23194,"tokens_out":35712,"duration_ms":289755,"significance":"If the mechanism is as claimed, this is a meaningful advance for federated continual learning: it removes the common task-ID-during-inference assumption via a lightweight subspace-relevance predictor, is replay- and generator-free, and is empirically well supported (three benchmarks; ablations over number of clients, thresholds, batch sizes, backbones, and task orders; standard deviations over seeds; per-task communication-cost tables). The honest discussion of the TIP mechanism's weaker regimes (20-split ImageNet-R, shared-classifier DomainNet) is a point in the paper's favor. The central issue is that the written mathematics does not support the stated mechanism: Eq. (4) applies the projection on the wrong side of the gradient relative to the invariance used in Eq. (3), and Appendix B's projector algebra is invalid for the basis constructed in Eq. (9). These are fixable in a revision, but until the formulas are corrected and matched to the implementation, the near-zero forgetting cannot be attributed to the stated orthogonal-projection mechanism.","major_comments":[{"comment":"The invariance claimed in Eq. (3) requires that subsequent updates leave previous-task pre-activations unchanged, i.e., ΔW̃_l^(t) a_l^(τ) ≈ 0 for every earlier-task activation a_l^(τ) in the span of Φ_l^(1:t-1), equivalently ΔW̃ Φ = 0; this is achieved by right-multiplying the update by (I - ΦΦ^T), because (I - ΦΦ^T)a = 0 for all a in col(Φ). Eq. (4), however, defines the left-multiplied update ∇W̃_k = (I - ΦΦ^T)∇W_k, which does not imply (I - ΦΦ^T)∇W_k Φ = 0: for Φ = e_1 and ∇W = [[1,2],[3,4]], (P∇W)Φ = (0,3)^T ≠ 0, so the pre-activation of a previous-task input changes by -η(0,3)^T. The expression in Eq. (4) is also dimensionally undefined when the layer's output dimension d_{l+1} differs from its input dimension d_l (the product Φ^T∇W requires d_l = d_{l+1}), which is the case for the final classifier (512 to 10) and at channel-transition layers of ResNet-18. Appendix B, Eq. (14), repeats the same left-multiplied definition. Thus, as written, the projection does not implement the orthogonal-projection mechanism from which Eq. (3) is derived, leaving the central forgetting-mitigation guarantee unsupported. Please correct the side of the multiplication (∇W̃ = ∇W - ∇W ΦΦ^T is the GPM-consistent form) and confirm which operator the released code applies; if the implementation matches Eq. (4) literally, the near-zero forgetting must instead be attributed to components such as head freezing and task routing, and the paper's mechanism claim would need to be substantially revised.","section":"Section 3.2, Eqs. (3)-(4); Section 4.1; Appendix B Eq. (14)"},{"comment":"The server-side aggregation in Eq. (9) appends residual columns U_k^(t) - Φ Φ^T U_k^(t) to the global basis Φ without renormalization. These residuals are orthogonal to the columns of Φ by construction, but they generically have norm strictly less than 1, so the assembled global Φ does not have orthonormal columns. The idempotence claim P² = P in Appendix B, Eq. (15), requires Φ^T Φ = I and therefore fails for the aggregated Φ; likewise, Eq. (6)'s 'projection' of activations no longer annihilates components in the span of the previous bases, since for a = Φc one gets (I - ΦΦ^T)a = Φ(c - Φ^T Φ c) ≠ 0 in general. The paper should normalize the appended bases or re-orthogonalize the global subspace (e.g., by an additional SVD on the server), and update Appendix B's algebra accordingly.","section":"Section 4.3, Eq. (9); Appendix B, Eq. (15)"},{"comment":"The conclusion of Appendix B that 'the convergence behavior of the algorithm is preserved' is not established by the argument given. The paragraph shows only that the projected stochastic gradient inherits the norm and variance bounds of the unprojected gradient (true when P is contractive, i.e., when the bases are orthonormal). But E[∇F̃_k] = P∇F_k is a biased gradient of the original objective, the fixed points of the projected dynamics (points with P∇F_k = 0) are not stationary points of the federated objective in Eq. (1), and no convergence rate or limit statement is provided for the projected objective either. The assumptions of Li et al. (2019) are transferred without transferring the theorem; a formal convergence statement, or a clear relegation of this paragraph to a heuristic motivation, is needed.","section":"Appendix B (convergence analysis)"},{"comment":"The protocol for the task-agnostic class-incremental evaluation is under-specified. In Figure 2a and Table 13 it is not described how each baseline performs inference when the task identity is unknown (e.g., classification through the full expanded head, per-task heads with a default routing rule, or a method-specific mechanism); Section 5.2 specifies a shared single classifier only for the DomainNet experiments. Because the headline margin (86.00 vs 41.80 on CIFAR100) depends on this protocol, the task-agnostic inference rule for every method should be stated explicitly, and a stronger task-agnostic variant of the most relevant baseline (e.g., LODE applied to FOT) would strengthen Table 4. Relatedly, Appendix C.2 should state whether the partial freezing scheme (freezing the first two residual blocks after task 1, and freezing previously learned output heads) is applied to all methods or only to FedProTIP, as this architectural choice can confound the forgetting comparison.","section":"Section 5, Figure 2, Table 13; Appendix C.2"}],"minor_comments":[{"comment":"Section 5.3 refers to the PEC/SCR/LODE comparison as 'Table 5' while that table is captioned 'Table 4', and the caption of the client-scaling table reads 'Table 3: Table 3: ...' with a duplicated label; the cross-references and captions should be corrected.","section":"Section 5.3, captions of Tables 3 and 4"},{"comment":"The caption of Table 9 mentions '20-Split DomainNet' and the caption of Table 11 says '6-split DomainNet' while its table header reads '20-Split DomainNet', although all DomainNet experiments in the paper use a 6-split configuration; the dataset-split labels should be made consistent.","section":"Appendix A.3, A.4 (Tables 9 and 11)"},{"comment":"Table 13 reports TARGET's task-agnostic average accuracy on 10-split CIFAR100 as 27.55 ± 0.89, while Table 10 reports 27.37 ± 1.00 for the same setting; the two tables should agree.","section":"Tables 10 and 13"},{"comment":"No ablation or guidance is provided for choosing m_s (the number of sampled activations), and the paper does not analyze how m_s and the threshold ε_l jointly determine whether the extracted bases span the dominant directions of each task's feature manifold; an ablation over m_s would substantially support the representativeness assumption on which both the forgetting mechanism and the task-ID references rely.","section":"Section 4.2"},{"comment":"Figure 1's caption writes the client vote as the argmax of the inner product ⟨ω̂, ω_k^(τ)⟩, whereas Eq. (11) defines the similarity as the cosine similarity with normalized vectors; the figure and the equation should be made consistent.","section":"Figure 1 and Eq. (11)"},{"comment":"The communication-cost table appears to count only newly extracted bases per task; the broadcast of the cumulative global bases Φ^(1:t) to all clients and the transmitted reference vectors should be itemized (or the asymptotic claim in Section 5.4 adjusted), and the units (MB vs MiB: the text says '48 MiB' while the caption says MB) should be harmonized.","section":"Section 5.4 and Table 14"},{"comment":"The privacy assertions ('preserving data privacy', 'privacy-preserving') in the abstract and contribution list are stronger than the discussion in Appendix D.2 supports, given that clients transmit subspace bases of intermediate activations; the claims should be qualified to match D.2's cautious framing.","section":"Abstract, Section 4, Appendix D.2"},{"comment":"Recent FCL methods discussed in related work (FedWeIT, TagFed, pFedDIL, AF-FCL) are not included in any experimental comparison; the state-of-the-art claim would be better supported by including them or by explicitly justifying their exclusion.","section":"Related work (Sections 2.1 and D.1)"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the left/right multiplication inconsistency between Eqs. (3) and (4). If the released code follows the standard GPM right-multiplication, this is a fixable presentation error and the empirical story is plausible; if the code literally matches Eq. (4), the claimed mechanism is not implemented and the near-zero forgetting would need a different explanation. I recommend asking the authors to confirm the implemented operator with a code excerpt. A second point to monitor is the symmetry of the experimental protocol (baseline inference rules in the task-agnostic setting and the partial-freezing scheme). The paper is within scope for this journal, the empirical study is broad, and I see no citation or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two quick things. FedProTIP targets a real problem—replay-free, task-agnostic federated continual learning—and the empirical results are impressively broad: three benchmarks, multiple heterogeneity levels, ablations on clients, thresholds, batch sizes, backbones, and task orders, all with standard deviations. The gains over FOT and other baselines are often large (86.00 vs 41.80 on 10-split CIFAR100 task-agnostic). The subspace-relevance task-ID predictor and the low-rank basis aggregation are genuinely new ingredients in this setting.\n\nHowever, the paper has a load-bearing math error. Eq. 4 projects gradients on the wrong side. The core bases Φ are extracted from input activations a_l ∈ R^{d_l × m}, so preserving previous tasks requires ΔW_l Φ = 0. That is enforced by right multiplication, ΔW ← ΔW (I − ΦΦ^T). The paper instead writes ΔW ← (I − ΦΦ^T) ΔW, a left multiplication, which does not imply ΔW Φ = 0 and so does not justify Eq. 3. Appendix B repeats the same mistake and also assumes Φ^T Φ = I, while Eq. 9 appends unnormalized residuals, so the stated projector is not idempotent either. This is not a cosmetic typo; it is the theoretical core of the forgetting-mitigation claim. If the released code uses the standard GPM right multiplication, the experiments may still be valid, but the paper as written doesn't support them.\n\nSmaller issues: the 'first to bring task-identity prediction into FCL' claim clashes with pFedDIL, which they themselves cite in Appendix D.1; and the abstract's 'up to 47%' framing hides the fact that TIP helps less on harder splits like 20-split ImageNet-R (though it still helps).\n\nOverall: the empirical work is solid and worth taking seriously, but the central equations need correction before publication. The convergence appendix is too shallow to count as theory and should be revised or cut.\n\nWho should read this: researchers in federated continual learning, especially those working on task-agnostic inference. If the implementation matches the intended projection, it's a useful method.\n\nRecommendation: don't desk-reject. Send to review, but with a referee who knows GPM and will check the side of the multiplication and the normalization.","headline":"The empirical case is strong, but the core projection equation is written on the wrong side of the weight matrix, so the paper's central forgetting-mitigation mechanism is not mathematically supported as stated.","tokens_in":23830,"tokens_out":4549,"would_cite":false,"duration_ms":38018,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedProTIP shows that federated continual learning can avoid catastrophic forgetting without replay memory, generative models, or task labels, by projecting updates onto the orthogonal complement of previously learned task subspaces and…","keywords":["federated continual learning","catastrophic forgetting","gradient projection","task-agnostic inference","task identity prediction","randomized singular value decomposition","class-incremental learning","subspace learning"],"falsifier":"Compare the subspace built from the random subset with the subspace built from the full set of a task's internal activity patterns; if the subset's basis misses a meaningful fraction of the directions the full basis spans, the orthogonal projection will not protect earlier tasks, and accuracy on those tasks should fall as soon as a new task is trained. Concretely, on 10-split CIFAR100, vary the subset size from tens to hundreds of samples and plot average accuracy: if accuracy stays flat, the assumption holds; if it collapses, the assumption fails.","tokens_in":22645,"feed_emoji":"🧠","tokens_out":8138,"duration_ms":67370,"temperature":0.7,"pith_summary":"FedProTIP claims that catastrophic forgetting in federated continual learning can be prevented without replay data, generative models, or knowing which task a test input belongs to. It does so by having each client extract low-rank \"core bases\"—the dominant directions of its current task's internal activations—and then projecting every subsequent local gradient update onto the orthogonal complement of all previously learned task subspaces. The same subspaces double as a task-identity predictor at inference: a test input's activation is compared, via cosine similarity, against per-task reference vectors, and the winning task routes the input to the right classifier head. If correct, this would make federated models that see an endless stream of tasks substantially more practical, and the paper reports large gains, e.g. 86.00% average accuracy versus 41.80% for the best baseline on 10-split CIFAR100 in the task-agnostic setting.","feed_headline":"86% vs 42%: replay-free projection wins task-agnostic continual learning","feed_subtitle":"Clients send only compact basis vectors, avoiding replay buffers, generators, and task labels at test time.","key_machinery":"The load-bearing object is the set of layer-wise core bases $\\Phi_l^{(1:t)}$, a low-rank orthonormal basis for the dominant subspace of each layer's activations across tasks, extracted by randomized SVD and aggregated at the server. From these bases the method builds the projection operator $P = I - \\Phi\\Phi^{\\top}$, which is applied locally to gradients, and the relevance metric $\\omega^{(\\tau,t)} = \\|U_L^{(t)}(U_L^{(t)})^{\\top} a_L^{(\\tau)}\\|_2$, which measures how strongly a final-layer activation aligns with a task's subspace. The projection preserves earlier representations; the relevance vectors, aggregated into reference vectors per client and matched by cosine similarity, supply task identity at inference. In one phrase, the engine of the paper is \"subspace as memory\": the same low-rank basis is used both to prevent forgetting and to route test inputs to the appropriate output head.","core_discovery":"On FedProTIP's own terms, the central discovery is that a federated gradient-projection scheme can be made communication-cheap and privacy-respecting by moving the orthogonal projection to each client and sending only aggregated core bases, and that these same bases enable task identity to be inferred at test time without any auxiliary classifier. Each client computes activations for a small random sample of its current-task data, subtracts their components lying in previously learned subspaces, and applies randomized singular value decomposition to obtain low-rank bases. The server agglomerates these bases into a global subspace. Training updates that would overwrite old representations are removed by orthogonal projection, while the final-layer activation's norm in each stored subspace forms a relevance vector compared against reference vectors to predict the task. The paper argues this achieves near-task-aware accuracy under task-agnostic class-incremental inference and consistent gains across heterogeneity levels, client counts, and dataset scales.","pith_inferences":["Inference: the relevance scores should also work as a novelty signal; a test input with low similarity to every stored reference vector could be flagged as out-of-distribution or belonging to an unseen task. The paper does not make this claim.","Inference: the method's reliance on a random subset of activations implies a testable scaling law: as the subset size grows, the sampled bases should approach the full activation SVD and forgetting should decrease. Measuring that curve would directly probe the paper's weakest assumption.","Inference: the majority-vote task predictor treats clients equally; weighting votes by client data volume or subspace confidence is a natural extension that could recover some of the accuracy lost on the 20-split ImageNet-R setting, where the paper reports smaller task-identity-prediction gains."],"forward_implications":["Replay-free and generator-free federated continual learning becomes viable: methods that store old examples or train generators to reproduce them are no longer needed to keep accuracy on earlier tasks.","Task-agnostic inference no longer requires a separate task classifier: the same subspaces that protect old tasks can route test inputs to the right head, yielding 86.00% versus 41.80% average accuracy on 10-split CIFAR100.","Forgetting stays near zero as tasks accumulate: final forgetting of 0.83% on CIFAR100 at $\\alpha=0.5$, and below 3% on DomainNet, compared with double-digit forgetting for most baselines.","Communication overhead shrinks as the task sequence grows: reported per-client per-task cost drops from about 9.8 MB to 0.3 MB, an order of magnitude below the 48 MB fixed cost of FOT.","The threshold $\\epsilon_l$ gives an explicit stability-plasticity dial: higher values favor preserving old tasks, lower values leave more room for new ones, with accuracy largely insensitive across 0.7–0.9."],"supporting_citations":[{"why":"Supplies the original gradient projection memory scheme: extracting core bases from activations and projecting updates orthogonally, which FedProTIP adapts to the federated setting.","marker":"(Saha et al., 2021)"},{"why":"FOT, the direct federated predecessor that collects high-dimensional client embeddings centrally; FedProTIP's local projection and compact bases are designed to avoid its communication and privacy costs.","marker":"(Bakman et al., 2024)"},{"why":"Theoretical decomposition of class-incremental learning into within-task classification plus task-identity prediction, motivating the paper's task identity prediction mechanism.","marker":"(Kim et al., 2022b)"},{"why":"FedAvg, the aggregation protocol and a principal baseline; FedProTIP keeps the same server aggregation while changing local updates.","marker":"(McMahan et al., 2017)"},{"why":"Provides the non-IID FedAvg convergence assumptions that FedProTIP's projection step is shown to preserve.","marker":"(Li et al., 2019)"},{"why":"Gradient-inversion attack result that motivates avoiding transmission of raw activations and gradients.","marker":"(Geiping et al., 2020)"}],"fun_headline_variants":["No replay, no task IDs: federated projection curbs forgetting","Federated continual learning: subspace projection beats replay","Task-agnostic FL: clients send bases, not data, to forget less","Replay-free federated learning with built-in task prediction","Project to orthogonal subspaces: continual learning without replay"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a small random sample of a task's internal activity patterns represents the whole task's feature directions well enough that the subspace built from it captures almost every direction later updates could disturb.","fun_headline_variants_meta":{"raw":{"variants":["No replay, no task IDs: federated projection curbs forgetting","Federated continual learning: subspace projection beats replay","Task-agnostic FL: clients send bases, not data, to forget less","Replay-free federated learning with built-in task prediction","Project to orthogonal subspaces: continual learning without replay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":2040,"prompt_tokens":953,"completion_tokens":1087,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1001}},"tokens_in":569,"tokens_out":1087,"duration_ms":9560,"temperature":1.0,"reasoning_tokens":1001,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:46:01.281903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the subspace built from the random subset with the subspace built from the full set of a task's internal activity patterns; if the subset's basis misses a meaningful fraction of the directions the full basis spans, the orthogonal projection will not protect earlier tasks, and accuracy on those tasks should fall as soon as a new task is trained. Concretely, on 10-split CIFAR100, vary the subset size from tens to hundreds of samples and plot average accuracy: if accuracy stays flat, the assumption holds; if it collapses, the assumption fails.","supporting_citations":[],"review_version":1}