{"id":"eeae0317-0059-4235-8a31-e4938e72a21d","arxiv_id":"2505.04575","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"KA-Prompt improves domain incremental learning by initializing new prompts from a greedily mined set of old ones and adding a cross-domain alignment loss, gaining about 4.7 average points over C-Prompt.","lead":"KA-Prompt is a new training method for models that learn one visual domain after another. It tries to fix mismatched 'knowledge components' in the prompts that such models use, and reports accuracy gains of about 4 to 5 percent on four standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KA-Prompt's Eq. 10 is only a cross-entropy loss on a weighted average of old and new prompts; it never constrains or measures component correspondence, so the empirical gains may stem from generic fusion regularization rather than the claimed component-wise alignment.","rationale":"I read the paper in good faith: it identifies a plausible limitation of C-Prompt, proposes a concrete two-phase mechanism, and supports it with consistent gains on four benchmarks plus ablations. The strongest claim, however, depends on the interpretation that the online loss enforces component-wise alignment. The reader's weakest assumption targets exactly this: Eq. 10 is cross-entropy on a weighted average, not a direct correspondence constraint. I agree that this is the most load-bearing point because both the problem identification and the methodological novelty rest on it. The shuffle ablation demonstrates sensitivity to component order, but it does not prove that the learned order corresponds to semantically meaningful, reusable knowledge components; it only proves that the fusion scheme is order-dependent. A random-prompt ablation would separate the alignment-specific effect from generic regularization. The Appendix-A fallback is an additional uncontrolled factor that should be reported. None of this invalidates the empirical contribution, so the reader's CONDITIONAL verdict remains appropriate; I recommend no change to the verdict.","tokens_in":16569,"tokens_out":5061,"duration_ms":58093,"concrete_test":"On ImageNet-R, rerun KA-Prompt with one change: in the Historical Prompt Online Aligning module, select the K old prompts for p_agn^x uniformly at random from the historical pool instead of by top-K query matching, while keeping the same weights (Eq. 8), the same initialization, and the same total number of gradient steps. If Avg-ACC stays within about 1 point of the reported 66.51, the online alignment loss is not specifically exploiting matched component knowledge and the component-alignment explanation is unsupported. If accuracy drops substantially, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that component-wise misalignment in prompts causes conflicting knowledge integration and that KA-Prompt fixes this by aligning prompt components. The load-bearing step is the online alignment loss. In Eq. 9, the 'aligned' prompt is a fixed elementwise weighted average of top-K old prompts and the new fused prompt; Eq. 10 then simply applies cross-entropy to that average. Nothing in these equations identifies or constrains which component slot carries which knowledge. The loss can be minimized by making the new prompt's components statistically compatible with the old prompts in aggregate, which is a regularizer or ensemble effect, without establishing the reusable component correspondence that the problem identification requires. The shuffle evidence (Fig. 2 and Fig. 7) does not resolve this: any model trained with an auxiliary loss on a particular elementwise fusion scheme will be sensitive to permuting components under that scheme, regardless of whether semantic slots are aligned. The Appendix-A fallback, which randomly interpolates two prompts when the cumulative score histogram is zero, is an uncontrolled branch whose frequency and effect on the reported numbers are never analyzed. Since the main contribution is the alignment mechanism, the proxy assumption in Eq. 10 is the most load-bearing point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KA-Prompt, a prompt-based domain incremental learning method built on the observation that per-domain prompts learned independently suffer from component-wise misalignment, so that fusing prompts across domains mixes unrelated knowledge components. To address this, the method (1) mines a set of relevant historical prompts via a greedy search and uses them to initialize new prompts, and (2) applies an online 'alignment' loss that trains the new prompts on a weighted elementwise fusion of old and new prompts. The method is evaluated on DomainNet, ImageNet-R, ImageNet-C, and ImageNet-Mix, reporting consistent gains over C-Prompt and other baselines, with ablations of the proposed modules, hyperparameter sensitivity, and overhead comparisons. The code is publicly available.","tokens_in":16846,"tokens_out":3678,"duration_ms":39644,"significance":"The paper addresses a real and practically relevant problem in prompt-based domain incremental learning, and the reported empirical gains are substantial and consistent across four benchmarks. The strengths of the manuscript are its thorough experimental evaluation, including per-domain results, module ablations, hyperparameter studies, and cost comparisons, as well as the release of source code. The proposed reusable-knowledge mining idea is interesting and could be useful beyond the specific alignment claim. However, the paper's central causal claim — that the gains come from component-wise prompt-knowledge alignment — rests on an indirect proxy: the alignment loss in Eq. (10) is a cross-entropy loss on a weighted elementwise fusion, and the manuscript does not directly measure or constrain component correspondence. The shuffle evidence is suggestive but not conclusive, because any model trained with an elementwise fusion auxiliary loss will be sensitive to component permutation. If the mechanism attribution is correctly established by additional experiments, the contribution would be solid; as written, the central mechanism remains undersupported.","major_comments":[{"comment":"The alignment loss L_agn is a cross-entropy loss on p_agn^x, which is an elementwise weighted average of old and new prompts. This loss does not identify, constrain, or measure which component slot carries which knowledge; minimizing it could simply make the new prompt statistically compatible with the old prompts in aggregate. Thus Eq. (10) functions as a fusion regularizer or ensemble objective, and the paper's core claim that it establishes componential knowledge alignment is not directly supported. Please provide a direct measurement of component-wise correspondence (e.g., a matching or correspondence metric between prompt components before and after training, or an analysis that component permutations destroy the learned structure in a way that a generic regularizer would not), or add a control experiment using an equally strong fusion regularizer that does not rely on specific component positions.","section":"Section 3.3, Eq. (10)"},{"comment":"The prompt-shuffle ablation is indirect and confounded. A model trained with an auxiliary loss on a specific elementwise fusion scheme will necessarily be sensitive to shuffling components under that scheme, even if no semantic alignment was ever established. The comparison with C-Prompt does not isolate the alignment mechanism, because C-Prompt is not trained with any elementwise fusion loss. A stronger test would be to compare KA-Prompt with a baseline that uses the same online fusion loss and hyperparameters but without the mined initialization, or to shuffle according to the mined correspondence versus random permutations and show that the alignment-consistent ordering is uniquely the best.","section":"Section 4.3, Fig. 7"},{"comment":"The fallback branch in the search algorithm, which randomly interpolates R=2 prompts when the cumulative score histogram is zero, is uncontrolled. The manuscript does not report how often this branch is taken, for which domains, or how the random interpolation affects the reported averages. Since the mined initialization is one of the two main mechanisms, this branch needs a frequency analysis and an ablation with a deterministic fallback (e.g., nearest prompt or a learned interpolation) to show that the random choice is not influencing the results.","section":"Appendix A"}],"minor_comments":[{"comment":"The subsection heading 'Reusable Knowledge Ming' is a typo; it should be 'Reusable Knowledge Mining'.","section":"Section 3.3, heading"},{"comment":"The word 'hypermeters' should be 'hyperparameters'.","section":"Section 4.1"},{"comment":"Both subfigures in Fig. 7 are labeled '(a)'; the second should be labeled '(b)'.","section":"Fig. 7"},{"comment":"In the line for obtaining top-K old prompts, the notation uses indices for the new prompt set (p_i^t) instead of the historical prompt set (p_i^{t-1}); please correct the indexing.","section":"Algorithm 2"},{"comment":"The phrase 'An base relation matrix' should be 'A base relation matrix', and the dimensions in the displayed equation should be checked for consistency with the definition of S0.","section":"Equation (4) and surrounding text"}],"recommendation":"major_revision","confidential_remarks":"The first page of the manuscript contains a header stating that it is from the Proceedings of the 41st International Conference on Machine Learning (PMLR 267, 2025). If this is not a formatting artifact, the journal should verify whether the submission constitutes prior publication. I did not consider this in the technical recommendation, which is based solely on the scientific content."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the empirical results are real: across four standard DIL benchmarks, KA-Prompt beats C-Prompt by 4 to 5.6 points, with small standard deviations, and the ablation study isolates the contribution of each module. Second, the central narrative—that component-wise misalignment between prompts is the problem and that their alignment loss fixes it—is not actually supported by the evidence they provide. Read it as a strong engineering contribution with an overclaimed mechanism, not as a clean scientific demonstration of alignment.\n\nThe genuinely new ingredients are the greedy mining of old prompts to initialize new ones (clearly better than just using the previous stage prompts, as in Wang et al. 2023a) and the adaptive weighting in the fusion loss. The paper is well written, the ablations are thorough, overhead is reported, and code is available. These are real positives, and the gains are consistent enough that I'd trust the numbers.\n\nThe soft spots are all about the alignment claim. Eq. 10 is just cross-entropy on an elementwise weighted average of old and new prompts. Nothing in that loss identifies or constrains which component slot carries which knowledge. The loss can be minimized by making the new prompt statistically compatible with the old prompts in aggregate, without ever establishing reusable component correspondence. The shuffle experiments don't rescue this: any model trained with a fusion loss on a particular component ordering will be sensitive to shuffling that ordering, regardless of whether semantic slots are aligned. The Appendix-A fallback—randomly interpolating two prompts when the cumulative score histogram is zero—is an uncontrolled branch whose frequency and effect on the reported numbers are never analyzed. That's a real gap, though probably minor in practice. Hyperparameters tau and lambda are tuned on ImageNet-R and then fixed; the paper doesn't show sensitivity across benchmarks, but the consistent gains across four datasets suggest this is not a big deal.\n\nNone of this invalidates the empirical result. But it shifts what the paper actually offers: a mining-plus-fusion recipe that improves prompt-based DIL, not a validated mechanism of componential alignment. The authors should either provide direct evidence of alignment (e.g., probing component correspondence or showing that the learned components become more stable across domains) or soften the language to match what Eq. 10 really does.\n\nThis paper deserves a serious referee. I'd send it out. For a reading group it's a good discussion piece, and if I worked in continual learning I'd cite it. My own recommendation after revision would be a borderline accept.","headline":"A solid empirical prompt-based DIL paper with consistent gains over C-Prompt, but the component-alignment story is more asserted than proven; the auxiliary loss is a regularizer that likely helps for other reasons.","tokens_in":17335,"tokens_out":1676,"would_cite":true,"duration_ms":19379,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that component-wise misalignment between domain-specific prompts causes conflicting knowledge fusion, and that KA-Prompt's mined initialization plus online alignment loss fixes it, gaining 4.25–5.59 accuracy points over…","keywords":["domain incremental learning","prompt-based methods","component-wise prompt alignment","reusable knowledge mining","greedy prompt search","online alignment","vision transformer"],"falsifier":"Run KA-Prompt to completion, then at inference apply a single fixed random permutation to the component positions of every prompt before fusion; the paper's own ablation shows accuracy drops under shuffling, so if the drop disappeared, or a shuffled variant matched or beat the unshuffled model, the claim that aligned component positions drive the gains would be falsified.","tokens_in":16373,"feed_emoji":"🧩","tokens_out":6714,"duration_ms":60625,"temperature":0.7,"pith_summary":"Domain-incremental learning systems that store knowledge in per-domain prompts risk mixing unrelated knowledge when prompts are averaged at inference time, because the components inside each prompt are learned independently and end up in different positions across domains. The paper argues this component-wise misalignment is an intrinsic limit of methods like C-Prompt, and that it also prevents new domains from reusing generalizable knowledge stored in old prompts. KA-Prompt addresses both problems: a greedy search mines a small set of old prompts whose knowledge is most relevant to the incoming domain, and those mined prompts initialize the new prompt set to create an initial alignment; an online alignment loss then keeps the new prompts' components consistent with matched old prompts as training proceeds. The result is a DIL method that improves average accuracy over C-Prompt by 4.25, 4.08, 5.59, and 5.00 points on DomainNet, ImageNet-R, ImageNet-C, and ImageNet-Mix respectively, with no extra learnable parameters.","feed_headline":"Component-aligned prompts add ~5 points in domain-incremental learning","feed_subtitle":"Aligned old-new prompt components beat the C-Prompt baseline on all four DIL benchmarks.","key_machinery":"KA-Prompt has two mechanisms. Reusable Knowledge Mining ($\\Psi_M$) computes a relation matrix between historical prompt keys and new-domain image features, then greedily adds prompts to a reusable memory by repeatedly picking the prompt that contributes the most uncovered new-domain knowledge, measured by a cumulative score histogram. Aligning-guided New Prompt Learning ($\\Psi_L$) initializes the new prompt set from that mined memory, then during training matches old prompts to the current input, weights each old prompt by how much its similarity falls below the new prompts' minimal matched score, and fuses old and new prompts into an aligned prompt whose cross-entropy loss $\\mathcal{L}_{agn}$ is added to the ordinary classification loss. The greedy selection supplies the initial component correspondence; the online loss is meant to preserve that correspondence as new prompts drift during training.","core_discovery":"The paper's central claim is that prompts are not monolithic: each prompt's $L_p$ components encode distinct types of knowledge, such as object-part features, and for prompts learned on different domains the same kind of knowledge tends to land in different component slots. When inference fuses the top-$K$ matched prompts by averaging, misaligned components merge unrelated knowledge into one vector, producing interference; random shuffling of component positions before fusion changes C-Prompt's accuracy, which the paper reads as evidence that the learned component order is far from optimal. KA-Prompt therefore enforces component-wise alignment: reusable old prompts are selected greedily by how much new-domain knowledge they uniquely cover, new prompts are initialized from them, and a weighted old-new fusion with a cross-entropy loss acts as an alignment-preserving regularizer while the new prompts learn. The paper reports that this yields consistent gains over C-Prompt on four DIL benchmarks, with the largest gains appearing in later domains as shared knowledge accumulates.","pith_inferences":["Editorial inference: the alignment loss is an indirect proxy; a direct component-correspondence metric, such as tracking how far each new component drifts from the old component it was initialized from, would isolate whether the mechanism is really alignment or generic regularization.","Editorial inference: the greedy coverage objective resembles submodular maximization, so formalizing prompt selection that way could yield approximation guarantees and transfer to other memory-and-retrieval selection problems.","Editorial inference: the same alignment principle might extend beyond DIL to task-incremental or class-incremental prompt pools, where reusable modules are also trained independently and fused later.","Editorial inference: a testable extension is to vary domain order so that some adjacent domains share little knowledge; if gains shrink, the mining step is doing the work, while persistent gains would point to the online alignment loss as a stronger regularizer."],"forward_implications":["If component alignment is the cause of the gains, then any prompt-based DIL method that fuses prompts across domains should benefit from a similar initialization-from-mined-prompts step, not just KA-Prompt.","Reusable knowledge can be transferred across domains without storing raw images, relying only on prompt keys and pre-trained features, which matters for privacy-sensitive applications.","Because no learnable parameters are added, the accuracy gains come from how prompts are initialized and regularized, not from a larger model.","The method's advantage grows with the number of seen domains, suggesting that alignment converts accumulated old knowledge into a reusable asset rather than a source of interference."],"supporting_citations":[{"why":"Provides the C-Prompt baseline and the compositional prompting fusion protocol that KA-Prompt builds on and compares against.","marker":"Liu et al., 2024a"},{"why":"Defines CODA-Prompt, an alternative prompt-fusion approach that KA-Prompt must beat in comparisons.","marker":"Smith et al., 2023"},{"why":"Introduces L2P, the prompt-pool paradigm for continual learning that underlies the matching and selection machinery.","marker":"Wang et al., 2022c"},{"why":"Supplies the previous-stage prompt initialization strategy used as the ablation baseline for the reusable knowledge mining module.","marker":"Wang et al., 2023a"},{"why":"Supplies the DomainNet benchmark used for the main accuracy comparison and the motivation shuffle study.","marker":"Peng et al., 2019"},{"why":"Supplies the ImageNet-R benchmark used in experiments and overhead comparisons.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the ImageNet-C corruption benchmark used in experiments.","marker":"Hendrycks & Dietterich, 2018"}],"fun_headline_variants":["Align prompt components to stop domain knowledge clashes","Prompt alignment beats random component fusion in DIL","KA-Prompt: align old and new prompt parts for better DIL","Greedy prompt reuse with alignment boosts domain learning","Component-wise prompt alignment tames domain interference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that training on a weighted average of old and new prompts forces matching knowledge components into matching positions, even though the method never directly constrains or measures which component in a new prompt corresponds to which component in an old prompt.","fun_headline_variants_meta":{"raw":{"variants":["Align prompt components to stop domain knowledge clashes","Prompt alignment beats random component fusion in DIL","KA-Prompt: align old and new prompt parts for better DIL","Greedy prompt reuse with alignment boosts domain learning","Component-wise prompt alignment tames domain interference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1546,"prompt_tokens":977,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":494}},"tokens_in":593,"tokens_out":569,"duration_ms":5351,"temperature":1.0,"reasoning_tokens":494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:24:27.245580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run KA-Prompt to completion, then at inference apply a single fixed random permutation to the component positions of every prompt before fusion; the paper's own ablation shows accuracy drops under shuffling, so if the drop disappeared, or a shuffled variant matched or beat the unshuffled model, the claim that aligned component positions drive the gains would be falsified.","supporting_citations":[],"review_version":1}