{"id":"3d9e9fb1-9ecc-419d-b20b-4c03fd81e1c9","arxiv_id":"2412.15571","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A frozen foundation model plus random Fourier features and linear discriminant analysis reaches joint-training-level accuracy in class-incremental learning without replay.","lead":"The authors propose KLDA, a method for class-incremental learning that applies random Fourier features and linear discriminant analysis to the frozen features of a foundation model. It reports accuracy matching the joint-training upper bound on several text and image benchmarks without storing any old data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No evidence of a test-set tuning protocol: σ and D are 'empirically determined' after seeing final accuracy, so the claim that KLDA matches joint training may not generalize to held-out tasks.","rationale":"I read the paper as claiming an empirical result: a frozen foundation model plus kernelized LDA statistics can match joint training without replay. The strongest evidence is the tables, not the theory. I agree with the reader that the theoretical justification for avoiding inter-task class separation is hand-wavy, and that the missing RanPAC baseline is a real omission. However, I do not think the Gaussian assumption is the single load-bearing point: LDA-style linear scoring can work well even when class-conditional distributions are not Gaussian, so a violation of that assumption would not automatically falsify the headline numbers. The load-bearing condition is instead that the reported accuracies are produced by a protocol that would also work for unseen tasks. The paper selects D and σ empirically, gives no validation procedure, and sweeps hyperparameters on the same datasets reported in the main table. That makes the central claim vulnerable to selection-on-test-set effects, which is exactly the kind of issue that should determine whether the 'matches the upper bound' statement is accepted as a general finding. The concrete test above would settle this: if the frozen hyperparameter configuration still matches joint training, the concern is resolved; if not, the conditional verdict should be maintained or tightened. I therefore recommend no change to the reader's CONDITIONAL verdict until such a test is run.","tokens_in":13811,"tokens_out":7306,"duration_ms":60874,"concrete_test":"Hold out the entire evaluation protocol: choose D=5000 and select σ using either a validation split of the first task only or a completely separate dataset (e.g., optimize σ on CLINC alone), then freeze these values and evaluate on Banking, DBpedia, and HWU without any further tuning. Recompute Table 1 under this protocol and compare each KLDA-E entry to Joint Fine-tuning. If any entry falls below the joint baseline by more than the reported standard error, the claim of matching the upper bound does not generalize beyond post-hoc tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that KLDA/KLDA-E reaches the joint-training upper bound without replay. The results rest on two hyperparameters, D and σ, and the paper's only guidance is that D=5000 and σ was 'empirically determined within range [10^-2, 10^-6] for each FM' (Implementation Details). No validation set or selection rule is described, and in true CIL the future tasks are unavailable when these choices must be made. Figure 1 then sweeps σ and D against the same datasets that appear in Table 1, so the reported numbers may be the best of a post-hoc sweep rather than the performance of a fixed configuration. The reader's Gaussian-assumption concern is related but not decisive: even if the RFF features are not Gaussian, LDA can still give good linear boundaries, so the empirical claim does not hinge on Gaussianity. What it does hinge on is that the reported accuracy is a fair, pre-registered prediction. The absence of any independent hyperparameter selection is the load-bearing gap: if σ and D were tuned on the reported test sets, the match to joint training is partly a selection artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KLDA, a class-incremental learning method that keeps a foundation model frozen, applies a radial-basis-function kernel approximated by random Fourier features (RFF) to the extracted features, and incrementally maintains class means and a shared covariance matrix, classifying via linear discriminant analysis (LDA). An ensemble variant, KLDA-E, averages softmax scores over several RFF draws. Experiments on four text intent/classification datasets (CLINC, Banking, DBpedia, HWU) and four image datasets (CIFAR-10/100, TinyImageNet, Stanford Cars) compare against fine-tuning, prompting, pseudo-replay, and prototype-based baselines, with joint fine-tuning of the frozen or fine-tuned foundation model as the upper bound. The central empirical claim is that KLDA(-E), without replay, reaches or exceeds the joint-training accuracy on most benchmarks.","tokens_in":14030,"tokens_out":6387,"duration_ms":59976,"significance":"If the reported numbers are robust, this is a practically valuable result: a simple, replay-free, non-parametric CIL method that leverages frozen foundation-model features and reaches the joint-training upper bound. The paper provides public code, repeated runs with random seeds, multiple language and vision backbones, and a consistent set of baselines, which are strengths. The main weakness is that the two hyperparameters of KLDA, the RFF dimension D and the kernel scale sigma, are selected after seeing the same benchmarks whose final accuracies are reported, with no held-out validation or explicit selection rule. Because the headline claim is parity with joint training, the risk of selection artifacts is the load-bearing issue. The theoretical justification for avoiding inter-task class separation is also asserted rather than proven.","major_comments":[{"comment":"The manuscript does not specify how sigma and D are selected. It states that D=5000 was found through \"empirical testing\" and that sigma is \"empirically determined within range [10^-2, 10^-6] for each FM,\" and Figure 1 sweeps these hyperparameters on the same datasets that appear in Tables 1-3. No held-out validation split, selection criterion, or pre-registered configuration is described. In a real CIL deployment, future tasks are not available at configuration time, so the reported parity with Joint Fine-tuning could partly reflect choosing hyperparameters after observing test-set outcomes. To support the central claim, the authors should either (a) select hyperparameters on a disjoint validation set and report the resulting accuracies, or (b) demonstrate that the Figure 1 curves are sufficiently flat that the choice is immaterial across all reported datasets. Without this, the main empirical claim is not yet established.","section":"Implementation Details and Figure 1"},{"comment":"The paragraph titled \"Theoretical Justification\" does not constitute a proof of the claim that KLDA avoids inter-task class separation. It asserts that because each class is represented as a Gaussian, \"effectively each task has only one class,\" so within-task prediction is always correct and the Gaussian serves as an OOD detector. This does not verify the conditions of Kim et al. (2022, 2023) and does not address the fact that the RFF-transformed features are not necessarily Gaussian; if they are not, the LDA decision boundary in Eq. (4) need not be optimal or safe against class confusion. If the theoretical argument is retained, it should be stated formally with explicit assumptions on the feature distribution and the sufficiency conditions from the cited work. Otherwise, the corresponding contribution claim should be weakened to an empirical observation.","section":"Theoretical Justification"},{"comment":"The paper positions KLDA as an extension of streaming LDA (Hayes and Kanan 2020) and mentions random projections and pre-trained models (McDonnell et al. 2023) in the introduction, but it does not include either SLDA or RanPAC as empirical baselines. Since the novelty of KLDA relative to plain LDA is the RBF/RFF feature transformation, a direct comparison against SLDA on the same frozen features is needed to isolate the contribution of the kernel component. Without such a comparison, the improvement over NCM and LDA reported in Table 1 cannot be attributed specifically to the kernelized features rather than to the incremental LDA mechanism itself.","section":"Related Work and Baselines"},{"comment":"The image experiments in Table 3 do not use task splits; the text says \"task splits are not required for this evaluation.\" This means the image results compare one-by-one class prototype accumulation against joint training, which is a valid feature-extraction benchmark but not a class-incremental learning protocol with a task sequence. The claim that KLDA is \"evaluated\" on image datasets should be qualified accordingly, or the experiments should be run under the same CIL protocol used for the text datasets, to avoid over-generalizing the image results to the incremental setting.","section":"Evaluation on Image Datasets"}],"minor_comments":[{"comment":"The references for DBpedia and HWU appear to be swapped: the paper cites Auer et al. (2007) for HWU, but that is the DBpedia paper, and it cites Liu et al. (2021b) for DBpedia, but that reference is the conversational-agent benchmark that is usually associated with HWU. Please correct these citations.","section":"Datasets"},{"comment":"NCM and LDA report zero variance (±0.00). This should be explained, for example by noting that their final prototypes and covariance are invariant to task order, so the only randomness across runs would come from the data split, which is held fixed.","section":"Table 1"},{"comment":"The text states that ensembles did not improve image results but does not explain why. A brief comment on this difference between text and image domains would help the reader interpret the role of the ensemble in KLDA-E.","section":"Table 3"},{"comment":"The term \"upper bound\" is used for Joint Fine-tuning. This is a standard CIL convention, but it should be described as the empirical joint-training baseline rather than a theoretical upper bound, since a fixed frozen feature extractor with a better classifier could in principle exceed it (as KLDA-E does on three datasets in Table 1).","section":"Conclusion / Terminology"},{"comment":"In the covariance update on line 17, the notation is slightly confusing because Ntotal is updated before being used in the coefficient Nprev/Ntotal. The formula is correct, but renaming the variables (e.g., T for total before update) would improve readability.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The hyperparameter-selection issue is the most important problem and should be pressed in the revision. If the authors can provide a held-out validation protocol or show that the Figure 1 plateau makes the choice immaterial, the paper could become acceptable. The swapped dataset references also suggest a need for careful proofreading. I do not see a fundamental flaw in the method itself, but the empirical claim is currently stronger than the evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper shows that a frozen foundation model plus random Fourier features plus LDA gets close to or above joint-training accuracy on several CIL benchmarks without replay. That is a real and useful result, if the numbers hold. The specific combination is new as a published recipe, though every ingredient is standard. The paper does a lot of things right: it tests across text and image datasets, uses several foundation models of very different sizes, compares against a reasonable set of baselines, reports repeated runs, and ships code.\n\nThe main soft spot is the hyperparameter story. The authors say D=5000 and sigma per FM were chosen through empirical testing, and Figure 1 sweeps sigma and D on the same datasets whose final accuracies appear in Table 1. No held-out validation set or selection rule is described. In real CIL you do not know future tasks when you pick sigma, so the reported numbers may be the best of a post-hoc sweep rather than the performance of a fixed configuration. The stress-test note is right that this is load-bearing. That said, the pattern is consistent across many datasets and many FMs, so I doubt the entire effect is a tuning artifact. Still, the paper needs a principled way to set sigma and D, or at least a sensitivity analysis on held-out tasks.\n\nThe other clear omission is RanPAC (McDonnell et al., 2023), which is cited but never included as a baseline. RanPAC is essentially random projections plus a prototype classifier on frozen features, very close in spirit. Leaving it out makes the comparison look incomplete. The theoretical justification is also hand-wavy: saying each class is a Gaussian makes within-task prediction trivially correct and the Gaussian an OOD detector does not really engage with the cited theory. That section could be cut or rewritten as intuition. Minor point: KLDA-E occasionally beats joint training by a small margin; that is plausible due to the regularizing effect of the shared covariance, but it deserves a comment.\n\nFor the right reader, this is a useful paper. It describes a simple, reproducible method with a strong empirical claim, and the code is available. It deserves a serious referee. I would ask for a validation protocol for hyperparameters, a RanPAC comparison, and a tighter theoretical section, but the core idea and the experiments are solid enough to warrant that effort rather than a desk reject.","headline":"Simple, strong empirical recipe for replay-free CIL on frozen features, but the headline claim leans on hyperparameters tuned on the test sets and misses the obvious RanPAC comparison.","tokens_in":14527,"tokens_out":1885,"would_cite":true,"duration_ms":19092,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A kernelized classifier over frozen foundation-model features matches the joint-training upper bound in class-incremental learning.","keywords":["continual learning","class-incremental learning","foundation models","random Fourier features","kernel methods","linear discriminant analysis","catastrophic forgetting","inter-task class separation"],"falsifier":"On a dataset whose classes are visibly multi-modal in the RFF-transformed feature space (for instance, each class consists of two widely separated clusters), run KLDA and compare final accuracy to joint fine-tuning; a large gap would falsify the shared-covariance Gaussian assumption, while near-equal accuracy would support it.","tokens_in":13622,"feed_emoji":"🎯","tokens_out":9254,"duration_ms":72032,"temperature":0.7,"pith_summary":"This paper claims that class-incremental learning, the setting where a model must keep recognizing old classes while learning new ones, can match the accuracy of joint training on all classes without replaying any old data. The proposed method, Kernel Linear Discriminant Analysis (KLDA), keeps a foundation model frozen and enhances its latent features with a random Fourier approximation of the RBF kernel, then incrementally updates a shared covariance matrix and class means. Classification is done by linear discriminant analysis in that kernelized space, so no network parameters are ever updated. On four text and four image benchmarks, KLDA reports final accuracies equal to or slightly better than joint fine-tuning of the same foundation model, which the field treats as the upper bound for class-incremental learning.","feed_headline":"Frozen foundation models plus kernel match joint-training accuracy","feed_subtitle":"KLDA classifies each class as a Gaussian in random Fourier space and matches the upper bound without replay.","key_machinery":"The machinery is Random Fourier Features (RFF), a Monte Carlo approximation of the Radial Basis Function kernel: $z(x) = \\sqrt{2/D}\\cos(Wx+\\beta)$ with $W$ drawn from $N(0, \\sigma^{-2}I)$ and $\\beta$ uniform on $[0, 2\\pi)$, so that $z(x_i)^\\top z(x_j)$ approximates $K(x_i,x_j) = \\exp(-\\|x_i-x_j\\|^2 / (2\\sigma^2))$. RFF turns the infinite-dimensional kernel map into a finite $D$-dimensional feature vector, avoiding the infeasible $N\\times N$ kernel matrix and making incremental mean/covariance updates possible. The other central object is the shared covariance matrix $\\Sigma$, updated online when each new class arrives, which together with the class means defines the LDA scoring rule. With $D=5000$, the method stores roughly 29.5M additional parameters for a BART-base model, dominated by the fixed RFF matrix and the covariance matrix.","core_discovery":"KLDA's central discovery is that the features of a frozen foundation model, which are only weakly linearly separable on their own, become sufficient for near-optimal class-incremental classification once they pass through a random Fourier feature map approximating the RBF kernel. In that transformed space the paper models every class as a Gaussian with a shared covariance matrix, so the optimal classifier is linear: weight vectors $w_m = \\Sigma^{-1}\\mu_m$ and biases $b_m = -\\frac{1}{2}\\mu_m^\\top \\Sigma^{-1}\\mu_m$. Because KLDA only accumulates first- and second-order statistics, catastrophic forgetting is impossible by construction, and the shared-covariance Gaussian per class provides a global decision rule that the paper argues eliminates inter-task class separation. The empirical claim is that this simple statistical model reaches the joint-training upper bound, with the ensemble version KLDA-E even exceeding it on three of four text datasets, while using no replay buffer and only a fixed random feature matrix.","pith_inferences":["If KLDA's results generalize, the practical recipe for CIL becomes: pick a strong frozen feature extractor, kernelize its outputs, and fit a Gaussian classifier; the large literature on replay, regularization, and prompt tuning may be unnecessary when such features are available.","The ensemble gain in KLDA-E suggests that increasing the random feature dimension $D$ might substitute for ensembling; comparing the accuracy of a single KLDA model with $D=25{,}000$ against KLDA-E with five models of $D=5{,}000$ would isolate the Monte Carlo approximation error.","Because KLDA only needs running means and a running covariance, it can be applied directly to online or streaming class-incremental learning where samples arrive one at a time, a setting the paper does not evaluate.","The shared-covariance Gaussian assumption is the most fragile part; extending KLDA to per-class covariance shrinkage or a mixture of Gaussians per class would test whether the upper-bound match survives when classes are strongly multi-modal."],"forward_implications":["Adding the RFF kernel map over frozen foundation-model features closes most of the accuracy gap that plain NCM and plain LDA leave to joint training, so the bottleneck in CIL is linear separability of features rather than forgetting itself.","No replay buffer is needed: KLDA stores only class means, a shared covariance matrix, and the fixed random feature matrix, so its memory does not grow with the number of tasks beyond the class means.","The method transfers across model families and sizes, from MiniLM and BART to Mistral-7b in text and DINOv2 in vision, suggesting the result is not tied to one foundation model.","Training is only moment accumulation, so KLDA trains in about 10 seconds on CLINC with BART-base, compared with 4 minutes for joint fine-tuning and 11-23 minutes for fine-tuning baselines.","Averaging probabilities across five random feature draws (KLDA-E) further improves accuracy, indicating that part of the remaining error is Monte Carlo variance of the kernel approximation."],"supporting_citations":[{"why":"Supplies the Random Fourier Features approximation that makes the RBF kernel computable in incremental CIL without storing the full kernel matrix.","marker":"Rahimi and Recht 2007"},{"why":"Defines Linear Discriminant Analysis, the classification rule KLDA uses with class means and the shared covariance matrix.","marker":"Izenman 2013"},{"why":"Establishes that good within-task prediction and out-of-distribution detection are necessary and sufficient for good CIL, the theoretical basis for KLDA's Gaussian-per-class design.","marker":"Kim et al. 2022"},{"why":"Extends the theoretical characterization of learnability in continual learning, supporting the claim that effective OOD detection per task avoids inter-task class separation.","marker":"Kim et al. 2023"},{"why":"Provides the streaming linear discriminant analysis baseline that KLDA extends by adding a kernel feature map.","marker":"Hayes and Kanan 2020"}],"fun_headline_variants":["KLDA: kernel trick lets frozen models beat forgetting","Random Fourier features cure catastrophic forgetting","No replay, no forgetting: kernel over frozen features","KLDA matches joint training without replay data","Frozen features plus RBF kernel hit upper bound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after the random Fourier transform, the features of every class are well described by one Gaussian with a common covariance matrix, so the linear discriminant scores are near-optimal; if real features are far from Gaussian or have very different covariances, the claimed natural separation between old and new classes is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["KLDA: kernel trick lets frozen models beat forgetting","Random Fourier features cure catastrophic forgetting","No replay, no forgetting: kernel over frozen features","KLDA matches joint training without replay data","Frozen features plus RBF kernel hit upper bound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1354,"prompt_tokens":981,"completion_tokens":373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":303}},"tokens_in":597,"tokens_out":373,"duration_ms":3527,"temperature":1.0,"reasoning_tokens":303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:17:52.011331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset whose classes are visibly multi-modal in the RFF-transformed feature space (for instance, each class consists of two widely separated clusters), run KLDA and compare final accuracy to joint fine-tuning; a large gap would falsify the shared-covariance Gaussian assumption, while near-equal accuracy would support it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Linear Discriminant Analysis, the classification rule KLDA uses with class means and the shared covariance matrix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that good within-task prediction and out-of-distribution detection are necessary and sufficient for good CIL, the theoretical basis for KLDA's Gaussian-per-class design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Extends the theoretical characterization of learnability in continual learning, supporting the claim that effective OOD detection per task avoids inter-task class separation."},{"cited_title":"L.; and Kanan, C","cited_arxiv_id":null,"evidence_quote":"Provides the streaming linear discriminant analysis baseline that KLDA extends by adding a kernel feature map."}],"review_version":1}