{"id":"c6749fdd-930e-431b-8cf3-d69e7bbc1e3a","arxiv_id":"2505.07690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AFA combines a contrastively trained shared LoRA adapter (AFFA) with an expanding multi-head LoRA mixture-of-experts (ABFA) to enable continual learning of CLIP while slightly improving its zero-shot transfer.","lead":"This paper introduces AFA, a continual learning method for vision-language models that uses a shared adapter to improve zero-shot recognition on unseen data while an expanding mixture-of-experts adapter preserves old knowledge. It reports state-of-the-art results on multi-domain task incremental learning benchmarks, including a small improvement over frozen CLIP's zero-shot accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AFA's claimed 0.9% gain over CLIP zero-shot is an artifact of comparing unlike dataset sets: CLIP's 69.4% excludes Aircraft, while AFA's 70.3% Transfer includes it; on the shared 10 datasets AFA is ~68.4%.","rationale":"Table 1's footnote creates a matched baseline for CLIP but not for AFA. The central claim in the abstract and Section 4.2 is that AFA 'surpasses the inherent zero-shot performance of CLIP in terms of transferability,' supported only by 70.3 vs 69.4. Once the set of datasets is aligned, the comparison reverses (68.4 vs 69.4 full-shot; 68.3 vs 69.4 few-shot). Because the paper's title, abstract, and contribution list are built around this result, the finding is load-bearing rather than cosmetic. This is more decisive than the reader's DDS-threshold concern: even if routing were perfect, the headline comparison is invalid as reported. The paper does contain useful components and AFA still appears to lead on Transfer among compared methods when Aircraft is excluded (68.4 vs BCL roughly 67.0 and DIKI roughly 66.2), so I would not call the entire framework vacuous; but the central 'beyond CLIP generalization' claim should be rejected or substantially reframed. Given the submission as is, the appropriate verdict is REJECT unless the authors can show, with code or an erratum, that the reported Transfer averages are computed on a common set and the gain survives.","tokens_in":22264,"tokens_out":21315,"duration_ms":179662,"concrete_test":"Recompute the Transfer averages in Table 1 and Table 2 on the identical 10-dataset set (exclude Aircraft from both the CLIP zero-shot row and every method row, as the footnote prescribes for CLIP). With the published AFA row values, full-shot Transfer becomes (67.9+45.7+54.8+71.3+88.5+64.4+89.7+64.7+67.0)/10 = 68.4% and few-shot Transfer becomes 68.3%, both below CLIP's 69.4%. If instead the authors maintain that Transfer should include Aircraft, report the CLIP baseline on the same 11 datasets and state explicitly that Aircraft is a training task, so this is not an unseen-domain comparison. Also report the same 10-dataset averages for DIKI and BCL to check whether the state-of-the-art comparison survives.","verdict_should_be":"REJECT","load_bearing_attack":"The headline claim that AFA's Transfer (70.3%) surpasses CLIP's zero-shot (69.4%) is not supported on a common evaluation set. In Table 1, the frozen-CLIP zero-shot average is computed with the footnote 'we exclude Aircraft when evaluating frozen CLIP's transfer capability,' i.e., averaging 10 datasets and omitting Aircraft's 24.3%. AFA's Transfer average, however, is computed over all 11 datasets and includes Aircraft at 88.6% — and Aircraft is the first training task in Order-I (Table 9). Summing AFA's other ten Transfer values in Table 1 gives 68.4%, not 70.3%, and in Table 2 the corresponding few-shot value is 68.3%; both are below CLIP's 69.4%. Thus the claimed 'beyond CLIP generalization' disappears as soon as both sides are restricted to the same datasets. The same mismatch affects the comparison with DIKI and BCL if their Transfer averages also include Aircraft. This is not a question of hyperparameter sensitivity; it is an arithmetic inconsistency in the paper's central quantitative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AFA (Against Forward and Backward Forgetting Adapter), a continual learning framework for vision-language models (VLMs) under multi-domain task incremental learning (MTIL). AFA combines two modules: AFFA, a task-shared LoRA adapter trained with contrastive loss to preserve and enhance zero-shot transfer, and ABFA, a mixture-of-experts adapter with multiple low-rank B matrices per expert to mitigate backward forgetting and improve few-shot learning. A domain distribution selector (DDS) routes test samples either to task-specific ABFA experts or to the AFFA path based on cosine similarity to task prototypes. Experiments are reported on MTIL, few-shot MTIL, CIL, and DIL benchmarks, with the central claim that AFA's Transfer score (70.3%) exceeds the frozen CLIP zero-shot average (69.4%).","tokens_in":22511,"tokens_out":9805,"duration_ms":73514,"significance":"If the claims were fully supported, the paper would make a meaningful contribution to continual learning of VLMs, particularly the idea of learning task-invariant information to improve zero-shot generalization and the multi-head LoRA MoE design for few-shot incremental learning. The reported improvements over prior MTIL methods on the Last metric (e.g., 87.2% vs. 85.0% for BCL in Table 1) are noteworthy, and the paper includes supplementary code and detailed ablations. However, the central novelty—exceeding CLIP's zero-shot capability—is currently not supported due to an inconsistency in the evaluation protocol. The method still warrants consideration, but the headline claim must be corrected and the experimental evidence strengthened.","major_comments":[{"comment":"The claim that AFA's Transfer (70.3%) surpasses frozen CLIP's zero-shot (69.4%) is based on an invalid comparison of different dataset sets. The footnote states that Aircraft is excluded when evaluating CLIP's transfer capability, but AFA's Transfer average includes Aircraft (88.6% in Table 1). Excluding Aircraft from AFA's Transfer values yields approximately 68.4% on the shared 10 datasets, which is below CLIP's 69.4% on the same 10 datasets. The same arithmetic inconsistency affects Table 2 (70.2% includes Aircraft 88.5%; the common-10 average is about 68.3%) and the Order-II results in Table 8. Consequently, the paper's headline conclusion that AFA 'surpasses the inherent zero-shot performance of CLIP' is unsupported by the reported data and must be corrected or withdrawn.","section":"§4.2, Table 1 and Table 2, Appendix Table 8"},{"comment":"Key hyperparameters of the proposed method are tuned directly on the benchmark test set, and the paper reports no multiple-seed runs or error bars. Specifically, the DDS threshold (0.75), the number of prototypes K (5), the number of experts (22), the top-k activated experts (2), and the LoRA rank (16) are all selected based on test-set performance curves (Fig. 3 and Appendix Fig. 4). Since the claimed margins over baselines are small (e.g., 1.8–2.2% in Average, 0.9% over CLIP), the absence of variance estimates makes it impossible to assess whether the reported improvements are statistically meaningful. The authors should provide results over multiple seeds and clarify which choices were made before or after seeing test-set performance.","section":"§4.3 and Appendix §9.4, Fig. 3 and Fig. 4"},{"comment":"The Transfer metric depends critically on the DDS routing threshold: unseen-domain samples must be correctly sent to the AFFA path for the transfer score to reflect genuine zero-shot generalization. The paper reports high task-selection accuracy (Table 11), but this accuracy is measured on the same benchmark used to tune the threshold, and the threshold selection in Appendix Fig. 4(b) is based on the MTIL test set. Without an evaluation on a held-out task sequence or a principled threshold-selection procedure, it is unclear whether the reported transfer ability is robust or an artifact of test-set tuning. This concern is especially relevant because the transfer gains are small in magnitude.","section":"§3.5, Table 11"}],"minor_comments":[{"comment":"There are typographical errors and redundancies: 'that that strengthens' appears in the abstract, and 'the final output of the AFFA module' in Eq. (10) should refer to the ABFA module.","section":"Abstract and §1"},{"comment":"The table formatting in the submitted text is inconsistent: several rows appear to have 11 numbers where 12 columns are expected (e.g., the AFA Transfer row in Table 1), making it difficult to verify the reported averages. Please ensure the final version presents all dataset-specific values and the average unambiguously.","section":"Table 1, Table 2, and Appendix Table 8"},{"comment":"The definition of the 'Average' metric is described as the mean of 'Transfer' and 'Last', but Table 1 reports 'Avg.' values that appear to be per-dataset averages rather than a simple mean of the two metrics. Please clarify the computation and state the formula explicitly.","section":"§4.1, Evaluation metrics"}],"recommendation":"major_revision","confidential_remarks":"The paper's central quantitative claim is invalid as reported because the comparison mixes different dataset sets. The authors should recompute all averages on a common evaluation set and either remove the 'beyond CLIP' claim or demonstrate it under a consistent protocol. If the claim is removed, the manuscript may still offer a useful incremental contribution to MTIL, but the evaluation should be strengthened with error bars and a clearer hyperparameter-selection procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTake a quick look, but set the headline claim aside. AFA — a task-shared LoRA trained with contrastive loss (AFFA) to fight forward forgetting, plus an expanding multi-head LoRA mixture-of-experts with per-task routers (ABFA) for backward forgetting — is a genuine departure from the current habit of routing unseen samples to frozen CLIP. The gains on seen-task retention (Last 87.2 vs BCL 85.0) and on few-shot MTIL are consistent across four benchmarks, the multi-head (1A/4B) ablation shows the intended effect, and the parameter savings over BCL (30.1M vs 59.8M) are real. The routing picker (DDS) works at ~99.7% selection accuracy. All of that deserves referee time.\n\nNow the soft spots. The paper reports no error bars or multiple seeds, and key hyperparameters (DDS threshold, prototype count, expert count, top-k, LoRA rank) are tuned on the benchmark's test set, so the absolute numbers are optimistic. The abstract promises code in the supplement; there is no code in the PDF.\n\nThe serious problem is the \"beyond CLIP\" claim in Table 1. Frozen CLIP's zero-shot average of 69.4 excludes Aircraft (footnote), because CLIP scores only 24.3 there. AFA's Transfer average of 70.3 includes Aircraft at 88.6. I checked the arithmetic myself: restrict both to the same ten datasets and AFA is about 68.4 — below CLIP's 69.4. If you instead include Aircraft in both, CLIP's average is 65.3 (which the paper's own Average column shows). Either way, the claim that AFA exceeds CLIP's inherent zero-shot capability is an artifact of averaging different dataset sets; it is not a result. The comparisons against BCL (68.9) and DIKI (68.3) are on the same eleven-dataset footing, so AFA's lead among MTIL methods probably stands — but the \"surpasses CLIP\" framing needs to be corrected or removed.\n\nThe method itself is coherent and the AFFA idea is worth engaging with. What's missing is code, repeated-seed statistics, a threshold-sensitivity analysis across task orders, and an honest transfer comparison on identical dataset sets. I'd send it to review, with a note that the headline claim must be fixed before acceptance. My own reading: the framework is a solid incremental step over BCL/DIKI even without the CLIP-surpassing claim.","headline":"AFA is a solid, practical MTIL framework, but the headline 'beyond CLIP' claim is an averaging artifact: restrict both sides to the same datasets and AFA (≈68.4) sits below CLIP (69.4).","tokens_in":23076,"tokens_out":15549,"would_cite":false,"duration_ms":129004,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a vision-language model can be taught new classification tasks one after another and come out better at recognizing domains it has never seen, by splitting the job between a task-invariant contrastive adapter and a…","keywords":["continual learning","vision-language models","zero-shot transfer","catastrophic forgetting","mixture-of-experts adapters","LoRA","multi-domain task incremental learning","parameter-efficient fine-tuning"],"falsifier":"Run the MTIL sequence again with the domain-selection threshold fixed at a value chosen without ever looking at the test set (for example, cross-validated on a held-out seen-domain split) and with the 11 datasets reordered so that visually similar domains are adjacent; if the Transfer score in either variant falls back below CLIP's 69.4% zero-shot reference, the claim that AFA enhances rather than merely routes around zero-shot generalization is undercut.","tokens_in":22043,"feed_emoji":"🧠","tokens_out":8632,"duration_ms":70366,"temperature":0.7,"pith_summary":"This paper takes aim at a specific failure of continual learning for vision-language models: when a model like CLIP is trained on a stream of new classification tasks, its pre-trained zero-shot ability erodes. Previous fixes handed unseen-domain test samples back to the untouched original CLIP, which stops the bleeding but cannot improve generalization. The paper proposes AFA, a two-track adapter framework in which a task-shared adapter trained with contrastive learning accumulates task-invariant knowledge to push zero-shot transfer above the original CLIP's score, while a mixture-of-experts adapter with multiple low-rank heads preserves each learned task. On the standard 11-dataset MTIL benchmark, AFA reports a transfer score of 70.3%, above the 69.4% zero-shot baseline, while retaining 87.2% accuracy on the last task.","feed_headline":"Continual learning that beats CLIP at its own zero-shot transfer","feed_subtitle":"A shared contrastive adapter turns each new task into a gain for unseen accuracy; a MoE adapter guards the old ones.","key_machinery":"The load-bearing mechanism is a pair of adapters mounted on a frozen CLIP backbone with a router between them. AFFA (against-forward-forgetting adapter) is one LoRA module shared across every task, trained on each incoming dataset with an image-text contrastive loss; its job is to accumulate task-invariant knowledge so that unseen-domain samples become easier, not harder, to recognize. ABFA (against-backward-forgetting adapter) is a dynamically expanding mixture-of-experts: a shared pool of experts, each a multi-head LoRA with one A matrix and several B matrices to limit interference among sub-concepts within a task, plus a task-specific router that is frozen once each task ends to stop forgetting. A parameter-free domain distribution selector compares a test sample's cosine similarity to K-means prototypes of each learned task and, if the top score falls below a threshold (0.75), sends the sample to AFFA, otherwise to the matching ABFA router.","core_discovery":"On the paper's own terms, the central discovery is that forward forgetting is not merely preventable but reversible: knowledge gathered during incremental learning can be turned into an asset for zero-shot generalization. The AFFA module, a LoRA adapter shared across all tasks and fine-tuned with an image-text contrastive loss on every sequential dataset, extracts task-invariant visual-linguistic structure, and at inference an unseen-domain sample is routed to this adapter rather than to the frozen CLIP. The result is a Transfer score of 70.3% on the 11-dataset MTIL benchmark, surpassing frozen CLIP's 69.4% zero-shot transfer by 0.9%, a first among MTIL methods. The companion ABFA module, a sparsely activated mixture of multi-head LoRA experts with progressively expanding task routers, carries the incremental side of the workload, reaching a Last accuracy of 87.2%, and the same benefits appear in 5-shot MTIL, class-incremental, and domain-incremental settings.","pith_inferences":["A test the paper does not run is to plot the Transfer score after each of the 11 tasks: if AFFA genuinely accumulates task-invariant knowledge, transfer should rise monotonically with task count, whereas a flat trajectory would suggest the gain is mostly benign regularization.","Since the routing threshold (0.75) and prototype count (5) were tuned with the benchmark test sets in view, re-running the protocol with those hyperparameters fixed on seen-domain validation data alone would show whether the 0.9-point transfer gain survives a stricter protocol.","The multi-head LoRA design is motivated by interference among implicit sub-concepts; measuring whether individual B heads specialize on identifiable sub-classes of fine-grained datasets would test that mechanism directly and predict where few-shot gains concentrate."],"forward_implications":["Each newly learned task can improve zero-shot transfer to unseen datasets instead of merely preserving it, directly contradicting the prior assumption that continual learning and generalization must trade off.","In the 5-shot regime the multi-head LoRA experts buy their largest gains on fine-grained and complex datasets such as Aircraft, DTD, and SUN397, making the framework a candidate for data-scarce continual learning.","The reported state-of-the-art scores come with lower training cost than the closest competitor: 30.1 million trainable parameters against BCL's 59.8 million, with 19.21% lower GPU memory.","The same two-adapter split carries over to class-incremental learning on CIFAR100 and TinyImageNet and to domain-incremental learning on DomainNet and CORe50, suggesting the mechanism is not tied to the MTIL protocol.","Ablations show the pair is complementary: a single plain adapter drops transfer by 24.4 points, ABFA alone loses only 0.1 point, and only the combination exceeds CLIP's zero-shot reference."],"supporting_citations":[{"why":"Supplies the frozen CLIP dual-encoder backbone that both adapters are built on, whose zero-shot transfer forms the baseline that AFA must exceed.","marker":"[30]"},{"why":"Defines the MTIL task, the two-order training protocol, and the Transfer, Average, and Last metrics used throughout the evaluation.","marker":"[53]"},{"why":"The primary state-of-the-art baseline (mixture-of-experts adapters with an auto-selector) that AFA extends and must beat on MTIL, few-shot MTIL, CIL, and DIL.","marker":"[51]"},{"why":"Establishes the competing paradigm of task-specific PEFT with dynamic expansion that AFA contrasts against by routing unseen samples to the task-invariant AFFA.","marker":"[38]"},{"why":"Supplies the LoRA low-rank adaptation mechanism used by both the AFFA module and the experts inside ABFA.","marker":"[16]"},{"why":"Provides the multi-head LoRA design with several B matrices per expert that motivates ABFA's few-shot capability.","marker":"[40]"},{"why":"Supplies the sparse top-k expert activation strategy that regulates the ABFA mixture-of-experts output to limit cross-task interference.","marker":"[5]"}],"fun_headline_variants":["AFA adapter turns incremental tasks into zero-shot wins","Reversible forgetting: continual adapter boosts CLIP zero-shot","Forward forgetting reversed to beat CLIP at zero-shot transfer","Continual VLM adapter surpasses CLIP's native zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The routing threshold that decides whether a test image comes from a seen or an unseen domain was chosen by tuning on the benchmark's test set, and the whole transfer claim depends on that threshold reliably sending genuinely unseen samples to the forward-forgetting adapter rather than accidentally to a task-specific expert.","fun_headline_variants_meta":{"raw":{"variants":["AFA adapter turns incremental tasks into zero-shot wins","Reversible forgetting: continual adapter boosts CLIP zero-shot","Forward forgetting reversed to beat CLIP at zero-shot transfer","Continual VLM adapter surpasses CLIP's native zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1349,"prompt_tokens":946,"completion_tokens":403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":562,"tokens_out":403,"duration_ms":4245,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:10:33.574554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MTIL sequence again with the domain-selection threshold fixed at a value chosen without ever looking at the test set (for example, cross-validated on a held-out seen-domain split) and with the 11 datasets reordered so that visually similar domains are adjacent; if the Transfer score in either variant falls back below CLIP's 69.4% zero-shot reference, the claim that AFA enhances rather than merely routes around zero-shot generalization is undercut.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP dual-encoder backbone that both adapters are built on, whose zero-shot transfer forms the baseline that AFA must exceed."},{"cited_title":"Preventing zero-shot transfer degradation in continual learning of vision-language models","cited_arxiv_id":null,"evidence_quote":"Defines the MTIL task, the two-order training protocol, and the Transfer, Average, and Last metrics used throughout the evaluation."},{"cited_title":"Boosting continual learning of vision-language models via mixture-of-experts adapters","cited_arxiv_id":null,"evidence_quote":"The primary state-of-the-art baseline (mixture-of-experts adapters with an auto-selector) that AFA extends and must beat on MTIL, few-shot MTIL, CIL, and DIL."},{"cited_title":"Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models","cited_arxiv_id":null,"evidence_quote":"Establishes the competing paradigm of task-specific PEFT with dynamic expansion that AFA contrasts against by routing unseen samples to the task-invariant AFFA."},{"cited_title":"Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA low-rank adaptation mechanism used by both the AFFA module and the experts inside ABFA."},{"cited_title":"Hydralora: An asymmetric lora architecture for efficient fine-tuning.Advances in Neural Information Pro- cessing Systems, 37:9565–9584, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the multi-head LoRA design with several B matrices per expert that motivates ABFA's few-shot capability."},{"cited_title":"Adamv-moe: Adaptive multi-task vision mixture-of- experts","cited_arxiv_id":null,"evidence_quote":"Supplies the sparse top-k expert activation strategy that regulates the ABFA mixture-of-experts output to limit cross-task interference."}],"review_version":1}