{"id":"ca9f7a76-1444-4744-bcbc-34c69e20eb24","arxiv_id":"2502.02909","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SPARC tunes tiny prompts in a PCA-compressed subspace of the task data, claiming almost no forgetting and very few trainable parameters for continual learning in LLMs.","lead":"The paper describes a way to teach a large language model new tasks without retraining the whole model or losing old skills. It compresses each task's training data into a few main directions, then tunes only a tiny set of prompt values inside that compressed space while the original model stays frozen.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never defines how the K-dimensional prompt matrix P is mapped to D-dimensional token embeddings, so the central subspace-tuning mechanism is unreproducible; the zero-forgetting result is also guaranteed by freezing per-task prompts rather than by PCA overlap.","rationale":"The reader's REJECT is well-founded. The most load-bearing gap is not the PCA expressiveness assumption per se; it is that the method's core computation is not specified. Section III.A.2 says soft prompts are a T×K trainable matrix prepended to input token embeddings, but LLM token embeddings are D-dimensional. The only PCA matrix defined is W ∈ R^{k×D} in Eq. (3), and no equation maps P into R^D. Consequently the reported results cannot be reproduced, and the parameter-efficiency claim (0.04% vs. 0.002% in Section IV.E) cannot be checked. A second, compounding issue is that Section IV.D says existing prompts remain frozen, so 'no forgetting' is a design consequence of separate frozen prompts, not evidence for subspace overlap or orthogonality. The PCA overlap analysis in Section III.B is therefore not validated by the forgetting numbers. I agree with the reader's rejection, though my emphasis differs from the reader's stated weakest assumption; the missing baseline and undefined forward pass are strong enough on their own. No code or formal verification is provided, so these internal gaps are not mitigated by artifacts.","tokens_in":8480,"tokens_out":5232,"duration_ms":51880,"concrete_test":"Re-derive the forward pass of Section III.A.2. Specifically, implement SPARC by mapping P ∈ R^{T×K} to the model's embedding dimension via the PCA matrix W (P W or P W^T, whichever makes dimensions match), and run the Section IV.C domain-incremental protocol on GPT-2. Then run the same protocol with ordinary prompt tuning (P ∈ R^{T×D}, one frozen prompt per task). If ordinary prompt tuning reproduces the reported zero/3%-forgetting numbers, or if neither implementation is consistent with the text's parameter counts, the central claim that PCA subspace overlap drives retention is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on training prompts in a K-dimensional PCA subspace while prepending them to D-dimensional token embeddings. Section III.A.2 defines the trainable prompt as P ∈ R^{T×K} and states each row is prepended to input token embeddings, but never defines the map from R^K to R^D. Section II.B introduces W ∈ R^{k×D} and projects data as X_k = X_c W^T, yet no equation applies W to P. If the intended map is P W (or P W^T), it should be stated; if P is instead meant to be D-dimensional, the 'lower-dimensional' claim and the reported parameter counts (0.04% vs. 0.002%) are inconsistent. Because this map is the mechanism by which subspace guidance reaches the model, its absence makes the empirical results unreproducible. Additionally, Section IV.D states existing prompts remain frozen, so zero forgetting in task-incremental learning is guaranteed by construction regardless of PCA overlap; standard prompt tuning with frozen per-task prompts would likely show the same. Thus the paper does not establish that the PCA/overlap machinery causes the reported retention.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPARC, a continual learning method for LLMs in which soft prompts are trained in a low-dimensional PCA subspace of the input embedding space. The method computes PCA on task embeddings, initializes prompts in the top-k principal subspace, measures cosine overlap between principal components of different tasks to decide whether to reuse an existing prompt, and otherwise initializes prompts in an orthogonal subspace. Experiments are reported on five domain datasets and five SuperGLUE tasks using GPT-2 and DeBERTa-base. The paper claims 0.04% trainable parameters, no forgetting in task-incremental learning, an average forgetting ratio of 3% in domain-incremental learning, and improved accuracy when combined with LoRA. Several central details of the method are missing or inconsistent, and the evaluation does not establish that the PCA/overlap machinery, rather than the frozen per-task prompt architecture, causes the reported retention.","tokens_in":8662,"tokens_out":4766,"duration_ms":48334,"significance":"The underlying idea—empirically identifying a low-dimensional subspace for prompt tuning and using subspace overlap to decide prompt reuse—is potentially interesting and would be a useful contribution if rigorously validated. The paper is clearly written and the proposed heuristic is easy to state. However, as submitted, the main claims are not reproducible: the map from the R^{T x K} trainable prompt to the D-dimensional token embedding space is never defined, the reported parameter-count percentages contradict one another, and the 'no forgetting' result is guaranteed by the architecture (frozen base model plus per-task frozen prompts) rather than by the PCA mechanism. The evaluation also lacks the standard prompt-tuning baseline needed to attribute the results to subspace awareness. If the authors can supply a precise method definition, corrected counts, and ablations against ordinary per-task prompt tuning, the framework could merit reconsideration; in its current form, the central claims are unsupported.","major_comments":[{"comment":"The prompt-to-embedding mapping is undefined, which makes the method unreproducible. Section III.A.2 defines the trainable prompt as P in R^{T x K} and states that each row is prepended to input token embeddings, but token embeddings are D-dimensional. Section II.B introduces W in R^{k x D} and projects data as X_k = X_c W^T, yet no equation applies W to P. If the intended map is P W or P W^T, it must be stated explicitly; if P is instead meant to be D-dimensional, the 'lower-dimensional' claim is void. Relatedly, the parameter counts are internally inconsistent: the abstract says 0.04% trainable parameters, Section IV.E reports less than 0.002% for T=10 and K=300, and Section III.A.2's example of T=10, K=100 gives 1,000 parameters, which is roughly 0.0008% of GPT-2's parameters rather than 0.04%. These numbers must be reconciled with the actual forward pass.","section":"III.A.2, II.B, IV.E"},{"comment":"The no-forgetting result in task-incremental learning is structurally guaranteed, so it cannot support the claim that PCA-based subspace alignment mitigates catastrophic forgetting. Section III.C.2 states that 'existing prompts remain frozen and accessible for inference,' and the base model is frozen throughout. In the task-incremental setup, prior-task accuracy is therefore unchanged by construction, regardless of PCA initialization, overlap analysis, or orthogonalization. A standard per-task prompt-tuning method with frozen prompts would exhibit the same zero-forgetting property. The paper needs a comparison to ordinary per-task prompt tuning and an ablation that isolates the effect of PCA-based initialization and reuse on downstream accuracy.","section":"III.C.2, IV.D"},{"comment":"The core decision mechanism—cosine overlap between PCA components with threshold tau—is never validated. Section III.B defines the reuse rule via overlap percentage and tau (e.g., 50%), but no experiment varies tau, reports the computed overlap percentages for the datasets used, or compares the reuse policy against simple alternatives such as always initializing a new prompt or always reusing the first prompt. The claim that subspace-guided prompt reuse produces the forward-transfer improvements in Section IV.C is therefore not supported. A concrete test would be to report task-pair overlap percentages and to compare SPARC's reuse decisions against random and trivial policies at fixed parameter counts.","section":"III.B, IV.B"},{"comment":"The experimental comparison does not establish superiority over standard prompt-based continual learning baselines. Figure 4 and Figure 5 compare PCA-based learning only against full fine-tuning and zero-shot inference; there is no standard prompt tuning baseline, no LoRA-only baseline, and no comparison to existing prompt-based continual learning methods (e.g., L2P, DualPrompt, ProgPrompt). In addition, the LoRA integration in Section IV.E is described only at a high level: the rank, placement, and how LoRA interacts with the PCA-projected prompts are not specified, making the '1% of parameters' claim unverifiable. The domain-incremental comparison in Figure 3(a) is against non-continual individual fine-tuning, which is not a continual learning baseline. Without these baselines and implementation details, the efficiency and accuracy claims are not established.","section":"IV.A, IV.E, Fig. 4"}],"minor_comments":[{"comment":"The figure cross-references are inconsistent: Section IV.C says the sequential training order is the x-axis of Fig. 2(a), and refers to results in Fig. 2, but Fig. 2 is the parameter-sweep plot; the domain-incremental results appear in Fig. 3. Please correct the cross-references.","section":"IV.B, IV.C"},{"comment":"The paper does not state the exact model sizes, embedding dimensions, optimizer settings, learning rates, or number of training epochs for either GPT-2 or DeBERTa-base, nor does it specify the number of PCA components K used in the main experiments. A reproducibility table is needed.","section":"IV.A"},{"comment":"The claim that the forgetting ratio 'consistently remains below 5%' is not backed by numerical values or error bars. Please report per-dataset forgetting numbers and standard deviations, especially since the abstract quotes an average of 3%.","section":"IV.C"},{"comment":"The statement that 'training cost is constant regardless of task count or model size' is misleading: per-task training cost may be constant, but total memory for stored prompts and PCA subspaces grows with the number of tasks. Please qualify this claim.","section":"IV.E"},{"comment":"Reference [19] is malformed ('M. G. Johannes Welbl, Nelson F. Liu, Crowdsourcing multiple choice science questions, 2017') and missing full publication details.","section":"References"}],"recommendation":"reject","confidential_remarks":"The missing prompt-space mapping and the internally inconsistent parameter counts are reproducibility blockers that are not curable by local edits within the current scope. The more substantive issue is that the headline no-forgetting result is an architectural artifact of frozen per-task prompts, so the paper does not currently demonstrate that the PCA/overlap machinery is responsible for the reported retention. If a revised submission provides the explicit forward pass, reconciles the counts, and adds a standard per-task prompt-tuning baseline plus ablations on the reuse policy, it could be reconsidered as a new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read SPARC this afternoon. Short version: the core idea is plausible, the write-up is not ready.\n\nWhat's actually new: prompting a frozen LLM in a PCA-reduced subspace, using cosine overlap between PCA components to decide prompt reuse, and orthogonalizing new prompts against old ones. That combination isn't in their references, and it's a reasonable heuristic. The experiments cover both task- and domain-incremental settings, and the LoRA variant shows a sensible accuracy/cost tradeoff.\n\nThe soft spots are real, and one is load-bearing. Section III.A.2 says the trainable prompt is P ∈ R^{T×K} and that each row is prepended to input token embeddings, but the model works in D dimensions. No equation maps K to D. The projection matrix W ∈ R^{k×D} from Section II.B is never applied to P. If the intended map is P W or P W^T, say so; right now the central mechanism is undefined, and the empirical results can't be checked.\n\nSecond, the 'no forgetting' result in task-incremental learning is guaranteed by the frozen-model/separate-prompt design. Existing prompts stay frozen; of course previous tasks don't degrade. Standard prompt tuning would almost certainly show the same. So that headline doesn't validate the PCA machinery.\n\nThird, the paper never compares to plain prompt tuning in the full embedding space. That is the natural baseline for a method claiming the subspace is what helps. The parameter counts are internally inconsistent: 0.04% in the abstract, less than 0.002% in Section IV.E, and the T×K=1,000 example in Section III.A.2 matches neither for GPT-2's size. Figure callouts also seem scrambled, which makes the evaluation hard to follow.\n\nWhat this paper is good for: a reader who wants a concise sketch of a subspace-based prompt-reuse idea and some initial numbers. It is not for someone who needs to reproduce or deploy the method.\n\nMy recommendation: don't desk-reject — the idea has enough substance that a referee could extract a solid method chapter if the authors add the missing map, run the standard-prompt-tuning baseline, and fix the numbers. But as submitted, it should not pass review; it needs major revision.","headline":"Plausible subspace-prompt idea, but the missing K-to-D map and absent prompt-tuning baseline make it unreproducible; needs major revision, not desk rejection.","tokens_in":9241,"tokens_out":4081,"would_cite":false,"duration_ms":34720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPARC makes LLMs learn tasks in sequence without forgetting","keywords":["continual learning","prompt tuning","PCA subspace","catastrophic forgetting","parameter-efficient fine-tuning","LoRA","task-incremental learning","domain-incremental learning"],"falsifier":"Train SPARC on pairs of tasks whose PCA subspaces have high cosine overlap but whose actual transfer accuracy is low (or low overlap but high transfer); if the overlap percentage does not track task-transfer accuracy, the prompt-reuse rule fails. A simpler check: on a single task, compare SPARC's top-k prompt against a full-embedding-space prompt of the same token count; if the full prompt clearly outperforms, the subspace expressiveness assumption is violated.","tokens_in":8248,"feed_emoji":"🧠","tokens_out":4288,"duration_ms":33850,"temperature":0.7,"pith_summary":"SPARC claims that continual learning in LLMs can be made almost free by treating each task as a low-dimensional subspace of the input embedding space. It trains only soft prompts, each represented in a PCA-reduced space, so the number of trainable parameters stays tiny (0.04% of the model). A cosine-similarity overlap test decides whether a new task can reuse an existing prompt or needs a fresh prompt initialized orthogonally to previous ones. The authors report zero forgetting in task-incremental learning, an average 3% forgetting ratio in domain-incremental learning, and full knowledge retention with improved accuracy when combined with LoRA. If these results hold, resource-constrained deployments could adapt LLMs to new tasks sequentially without storing old data or retraining the base model.","feed_headline":"Prompt-PCA method adds tasks to an LLM with zero forgetting","feed_subtitle":"SPARC tunes prompts in a compact subspace and reuses old ones, updating just 0.04% of the model's parameters.","key_machinery":"The load-bearing object is the per-task prompt subspace defined by PCA: a trainable prompt matrix P of size T×k is mapped through the task's top-k principal components instead of living in the full embedding space. Two geometric operations carry the method: cosine overlap percentage between principal components of different tasks decides prompt reuse, and orthogonal projection of the new task's embeddings onto the complement of previous subspaces decides where to initialize a novel prompt. Together they keep the base LLM frozen, so the only trainable parameters are the T×k prompt entries.","core_discovery":"The paper's central claim is that catastrophic forgetting can be avoided by aligning each new task's prompt with the geometry of previously learned prompt subspaces. Concretely, SPARC computes PCA on the task's input embeddings, keeps the top k principal components as the prompt's working space, and measures overlap between tasks by cosine similarity of these components. When overlap passes a threshold, the existing prompt is reused with light fine-tuning; when it does not, the new prompt is projected into the orthogonal complement of all earlier subspaces before training. The authors report that this scheme achieves 97% backward transfer in domain-incremental settings and no forgetting in task-incremental SuperGLUE settings, while fine-tuning only 0.04% of parameters, and that adding LoRA (1% of parameters) preserves all prior knowledge and improves accuracy.","pith_inferences":["If the overlap-to-reuse link is causal rather than correlational, the same PCA-overlap test could be used at inference time to select which stored prompt to load for an unlabeled input, turning the method into a prompt router.","The paper does not report a systematic sweep over the overlap threshold τ or the component count k; a natural extension would be to measure how accuracy and forgetting vary along those axes, and whether the optimal threshold transfers across model families.","The claim that prompt expressiveness is preserved in the top-k subspace could be stress-tested by comparing SPARC against full-space prompt tuning on tasks whose discriminative information lives in low-variance directions; PCA's variance-maximizing choice might then miss the signal.","Because the method never touches base weights, it could be combined with other frozen-model safeguards such as output constraints or input preprocessing without interference."],"forward_implications":["Task-incremental continual learning can, according to the paper, be done with no forgetting and with only 0.04% of the model's parameters trainable.","Domain-incremental learning across healthcare, science, oceanography, and general knowledge retains about 97% of prior accuracy, with average forgetting of 3%.","Because prompts are small and stored per task, the framework's training cost stays constant as the number of tasks grows.","The PCA + LoRA hybrid offers a tunable accuracy-cost tradeoff, using 1% of parameters while preserving all prior knowledge.","Forward transfer appears naturally: training on a healthcare domain improves later performance on scientific literature via shared subspace structure."],"supporting_citations":[{"why":"Supplies the prompt-tuning setup that SPARC builds on, training soft prompts while freezing the LLM.","marker":"[14]"},{"why":"Provides the PCA formulation used to define task subspaces and the transformation matrix.","marker":"[15]"},{"why":"Documents catastrophic forgetting in LLMs, the problem SPARC targets.","marker":"[5]"},{"why":"Introduces LoRA, which SPARC integrates for the hybrid PCA + LoRA variant.","marker":"[10]"},{"why":"Supplies the SuperGLUE benchmark and datasets used for task-incremental evaluation.","marker":"[23]"},{"why":"Provides PubMedQA, a biomedical question-answering dataset in the domain-incremental setup.","marker":"[18]"},{"why":"Provides SciQ, the scientific-reasoning dataset used in domain-incremental evaluation.","marker":"[19]"},{"why":"Provides TriviaQA, the general-knowledge dataset used in domain-incremental evaluation.","marker":"[21]"}],"fun_headline_variants":["SPARC: zero-forgetting LLM task addition via PCA prompt subspaces","Subspace-aware prompts let LLMs learn new tasks without forgetting","PCA prompt reuse keeps LLM knowledge intact while tuning 0.04%","SPARC: reuse prompt subspaces to add tasks to an LLM with no forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a prompt trained in the top-k PCA subspace of a task's input embeddings is as expressive as one trained in the full embedding space, and that cosine overlap between PCA components reliably predicts whether a prompt can be reused for a new task; the paper does not validate this link with transfer-accuracy experiments.","fun_headline_variants_meta":{"raw":{"variants":["SPARC: zero-forgetting LLM task addition via PCA prompt subspaces","Subspace-aware prompts let LLMs learn new tasks without forgetting","PCA prompt reuse keeps LLM knowledge intact while tuning 0.04%","SPARC: reuse prompt subspaces to add tasks to an LLM with no forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000377,"raw_usage":{"total_tokens":1992,"prompt_tokens":917,"completion_tokens":1075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":993}},"tokens_in":533,"tokens_out":1075,"duration_ms":10146,"temperature":1.0,"reasoning_tokens":993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:38:57.470741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SPARC on pairs of tasks whose PCA subspaces have high cosine overlap but whose actual transfer accuracy is low (or low overlap but high transfer); if the overlap percentage does not track task-transfer accuracy, the prompt-reuse rule fails. A simpler check: on a single task, compare SPARC's top-k prompt against a full-embedding-space prompt of the same token count; if the full prompt clearly outperforms, the subspace expressiveness assumption is violated.","supporting_citations":[{"cited_title":"Superglue: A stickier benchmark for general- purpose language understanding systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the SuperGLUE benchmark and datasets used for task-incremental evaluation."}],"review_version":1}