{"id":"79ab7d53-a513-4bec-b004-c4e3645f1159","arxiv_id":"2502.08075","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Learning new knowledge first and then forgetting selected classes outperforms the reverse order for a pretrained vision model, on classification, segmentation, and detection.","lead":"Knowledge Swapping is a new task: a pretrained vision model forgets user-specified classes, keeps old knowledge, and learns new classes in one update. The paper argues that learning new knowledge first and then forgetting is better than the reverse order, and demonstrates this on classification, segmentation, and detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported Learning-Before-Forgetting advantage is confounded with which operation is last; the two-stage comparisons end with different objectives, and the feature-hierarchy evidence is too weak to rule out a trivial final-phase effect.","rationale":"The Pith reader's weakest assumption correctly identifies the unquantified feature-hierarchy claim as load-bearing for the paper's mechanistic story. I agree that Discovery-I needs quantitative support, but I think an even more basic threat sits one level earlier: the headline empirical comparison is between two pipelines that end with different objective functions. L→F's final phase minimizes L_forget, so Accf=0 is built into the protocol; F→L's final phase has no forgetting term, so the forgetting set can legitimately reappear. This does not make the recipe useless—if the task is to end with a model that has forgotten, the recipe 'learn first, unlearn last' is a valid practical prescription. However, it means the experimental design cannot distinguish the proposed feature-hierarchy explanation from the trivial account that the last phase's objective dominates the final metric. The F→L→F row in Table 2 is the paper's own strongest evidence for this concern: after F→L, one additional forgetting phase brings mIoUf back to 0.08–0.39, close to L→F, suggesting that the initial order is not decisive once the final phase is controlled. The neutral-phase test I propose would settle whether order itself matters. The reader's verdict of CONDITIONAL remains appropriate; the paper should be required to add this control and to provide per-layer statistics with seeds for Discovery-I before the mechanistic generalization is accepted.","tokens_in":12225,"tokens_out":11696,"duration_ms":109295,"concrete_test":"Run a matched three-phase experiment on the Table 1 classification setting: (A) Learn → Forget → Neutral and (B) Forget → Learn → Neutral, where Neutral is a fixed number of retention-set-only fine-tuning steps with identical hyperparameters. If (A) keeps Accf ≈ 0 while (B) yields Accf well above 0 after the common final phase, the order effect is genuine; if both converge to similar Accf, the two-phase tables are a final-phase artifact. As a quantitative check on Discovery-I, report mean |ΔW| per Transformer block with standard errors over at least three seeds for the learning and forgetting phases, and test for monotonic shallow-to-deep vs deep-to-shallow structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Tables 1–3, L→F always ends with the forgetting phase, whose loss (Eq. 8) directly drives Accf/mIoUf to ≈0, whereas F→L ends with the learning phase, whose loss (Eqs. 5–7) contains no forgetting term. The rebound of forgetting accuracy after F→L is therefore expected by construction; it does not require the directional feature hierarchy of Section 3.2. The paper's own F→L→F rows in Table 2 show that adding one final forgetting phase after F→L restores forgetting mIoU to 0.08–0.39, i.e., the order of the first two phases matters little once the pipeline is forced to end with forgetting. Discovery-I, the mechanistic claim that learning is low-to-high and forgetting is high-to-low, is inferred solely from visual inspection of L2-norm curves in Figures 2 and 6, with no layer-wise statistics, no error bars, and no test on the classification or detection models of the main tables. Thus the central non-tautological claim—that learning should precede forgetting because of feature hierarchy—is not yet supported; the results are consistent with the simpler rule 'put the unlearning phase last.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Knowledge Swapping, a task in which a pretrained model must simultaneously learn new knowledge, retain existing knowledge, and forget user-specified knowledge. The authors propose a two-stage pipeline built on LoRA and group sparse regularization, and claim that performing the learning phase before the forgetting phase ('Learning Before Forgetting') yields substantially better results than the reverse order. The claimed explanation is a feature-hierarchy discovery: incremental learning progresses from low-level to high-level features, while forgetting proceeds from high-level to low-level features. The paper validates the strategy with experiments on ImageNet-100 classification, COCO-based object detection, and ADE20K-based semantic segmentation, and releases source code.","tokens_in":12431,"tokens_out":5010,"duration_ms":39604,"significance":"If the central claim is correct, the paper offers a simple and practically useful design principle for selective model editing: when swapping knowledge, train the new task before unlearning the old one. The task formulation itself is clear and the experimental sweep across three vision tasks is a strength, as is the public code release. However, the paper's key non-tautological claim—that the order matters because of a directional feature hierarchy—is currently undermined by a comparison that confounds order with which objective is optimized last, and by evidence for the hierarchy that is only qualitative. The benchmark is therefore not yet established to the standard needed for a strong claim.","major_comments":[{"comment":"The central comparison L→F versus F→L is confounded because the two sequences end with different optimization objectives. In L→F, the final phase minimizes Eq. (8), which directly drives the forgetting-set metric toward zero; in F→L, the final phase minimizes Eq. (7), which contains no forgetting term. The rebound of mIoUf after F→L is therefore expected even without any feature-hierarchy effect. This is confirmed by the F→L→F rows of Table 2, where adding a single final forgetting phase reduces mIoUf to 0.08–0.39 across the four datasets, comparable to the L→F rows. The large gap in Tables 1–3 between L→F and F→L thus does not establish the Learning Before Forgetting principle; it is consistent with the simpler rule that the unlearning phase should be last. Please add conditions that hold the final phase fixed while varying the order, or otherwise isolate the order effect.","section":"§5.3, Table 2"},{"comment":"Discovery-I is the sole mechanistic basis for the Learning Before Forgetting principle, but it is inferred from visual inspection of L2-norm and gradient curves (Figures 2, 4, and 6) on three semantic-segmentation settings. There are no quantitative layer-wise statistics, no error bars, and no analogous measurement on the classification or detection models used in Tables 1 and 3. The claim that learning progresses from low-level to high-level features and forgetting from high-level to low-level is therefore not tested on the models where the main results are reported. Please provide per-layer statistics (e.g., mean and variance of weight updates relative to pretrained weights) for all three task families, and check whether the observed pattern could be an artifact of the group-sparse LoRA regularization rather than an inherent property of learning and forgetting.","section":"§3.2, Discovery-I"},{"comment":"All results are single runs with no seeds or error bars. For a benchmark paper, variance estimates across at least three seeds are necessary to assess whether the observed margins are repeatable and whether the per-task hyperparameter choices (α, β, and BND reported in §5.1) materially affect the conclusions. Without such estimates, the quantitative claims in Sections 5.2–5.4 are not yet established to the standard expected for a benchmark.","section":"§5.1–§5.4, Tables 1–3"}],"minor_comments":[{"comment":"The heading 'Sparse Constriant' should be 'Sparse Constraint'.","section":"§4.3"},{"comment":"The word 'whiestimatetes' in the related-work paragraph should be 'which estimates'.","section":"§2.2"},{"comment":"The notation in the tables, e.g., 'L− →F', is hard to parse; please use a consistent arrow notation and explain it in the captions.","section":"Tables 1–3"},{"comment":"The term 'knock-on feature hierarchy' is non-standard; consider renaming it to something like 'feature-hierarchy interplay' for clarity.","section":"§3.2"},{"comment":"Oxford-IIIT Pet appears as a learning set in both the classification and segmentation setups; please clarify whether the splits and data composition differ between the two tasks.","section":"§5.1"},{"comment":"The paper surveys many continual-learning and machine-unlearning methods but includes none of them as experimental baselines; adding at least one adapted baseline would make the proposed benchmark more informative.","section":"§2 and §5"}],"recommendation":"major_revision","confidential_remarks":"The confound identified in Major Comment 1 is addressable with additional experiments, so I do not recommend rejection. However, the current manuscript overclaims the causal role of the feature hierarchy; the authors should be encouraged to either provide quantitative evidence for Discovery-I or soften the mechanistic claims to a heuristic. The task definition may be a conceptual contribution even if the method itself is simple."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper on Knowledge Swapping is worth a look. It proposes a new task—simultaneously learn new classes, forget specified classes, and retain the rest on a pretrained model—and backs it with consistent experiments across classification, segmentation, and detection. The main empirical result, that learn-then-forget beats forget-then-learn, is clear-cut and large in magnitude. That alone is a contribution: the task formulation and benchmark are genuinely new and the code is promised.\n\nThe problem is the paper's interpretation. The authors argue that learning goes from low-level to high-level features while forgetting goes high-to-low, and therefore learning first is better. But their own Table 2 contains a control that quietly undermines this. In the F→L→F row, they add a final forgetting phase after the forget-then-learn sequence, and the forgetting mIoU drops back to near zero. That means the initial order barely matters; what matters is that the last phase is forgetting. The L→F advantage is exactly what you'd expect if the objective of the final phase directly drives the forgetting metric. In F→L the final phase is learning, and the learning loss has no forgetting term, so forgetting accuracy rebounds. That is a confound, not evidence for a feature hierarchy.\n\nThe feature-hierarchy \"Discovery-I\" is itself weak: it comes from visual inspection of L2-norm curves, with no layer-wise statistics, no error bars, and no check on the classification or detection models. The paper also doesn't compare against any existing continual learning or unlearning method, reports single runs without seeds, and tunes hyperparameters per task without sensitivity analysis.\n\nNone of this kills the practical value of the task or the benchmark. A practitioner who wants to swap knowledge on a pretrained model can follow \"learn then forget\" and get good numbers—that's useful. But the mechanistic story should be dropped or heavily qualified. The paper needs a major revision that compares L→F against F→L→F in every setting, reports variance, and tests the hierarchy quantitatively. It deserves a serious referee, but not as is.\n\nI'd bring it to a reading group to discuss the confound, and I'd probably cite the task definition in future work on unlearning. Send it to review with a strong request for revision.","headline":"A useful new task and benchmark, but the claimed 'learning before forgetting' principle is confounded with which phase is last, and the feature-hierarchy story is not supported by the paper's own control.","tokens_in":13001,"tokens_out":3501,"would_cite":true,"duration_ms":28613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Knowledge Swapping, a task that forgets specified classes, keeps the rest, and learns new ones, and argues the reliable recipe is to learn first and forget second.","keywords":["Knowledge Swapping","machine unlearning","continual learning","Learning Before Forgetting","feature hierarchy","LoRA","group sparse regularization","selective forgetting"],"falsifier":"Measure per-block parameter-change statistics during learn-only and forget-only runs on the classification and detection models used in the paper's main tables: if learning updates are not concentrated in early blocks or forgetting updates are not concentrated in late blocks, the directional hierarchy that motivates the learn-first order is falsified.","tokens_in":11994,"feed_emoji":"🔁","tokens_out":5867,"duration_ms":43935,"temperature":0.7,"pith_summary":"The paper proposes Knowledge Swapping, a task in which a pretrained model must forget user-specified knowledge, keep essential knowledge, and absorb new knowledge in one update. It claims the order of the two phases decides success: train the new task first, then erase the forget-set, rather than the intuitive forget-then-learn. The reason offered is a directional feature hierarchy: incremental learning builds from low-level to high-level features, while forgetting travels from high-level semantics down to low-level features. If the claim holds, any knowledge-editing pipeline can be made more reliable simply by sequencing its fine-tuning and unlearning stages, with forget-set accuracy dropping to near zero while retention and new-task accuracy stay high.","feed_headline":"Learn first, forget later: recipe for reliable model knowledge swaps","feed_subtitle":"New benchmark: training new classes first, then erasing old ones, keeps forgotten accuracy near zero.","key_machinery":"The load-bearing object is the directional feature hierarchy, operationalized through low-rank adaptation matrices $\\Delta W = AB$ attached to the FFN linear layers of each transformer block, with group sparse regularization on $\\lVert A_k \\rVert_F^2 + \\lVert B_k \\rVert_F^2$ to retain only needed modules. The forgetting phase uses a boundary-constrained loss $\\mathcal{L}_{\\text{forget}} = \\mathrm{ReLU}(\\mathrm{BND} - \\mathcal{L}(f(X_f),Y_f))$ so erasure is stabilized while retention and learning losses continue. The machinery converts the ordering insight into a concrete two-stage update: learn low-level features before high-level erasure, so the forgetting pass stays confined to semantic layers and cannot be undone by later low-level changes.","core_discovery":"The central discovery is the directional feature hierarchy: in the studied transformer models, incremental learning progresses from low-level representations to higher-level semantic features, whereas targeted forgetting moves from high-level semantics down to low-level features. From this the paper derives the Learning Before Forgetting principle, a two-stage procedure that first fine-tunes on the learning set (with retention-set supervision) and only then runs selective forgetting. Empirically, this single ordering choice drives forget-set accuracy, mIoU, or mAP to near zero across image classification, object detection, and semantic segmentation, while the reverse order lets forgotten classes rebound after the learning phase because low-level changes invalidate the earlier high-level erasure.","pith_inferences":["If the directional hierarchy is a general property of transformer training, the learn-then-forget ordering should extend beyond low-rank adaptation to other parameter-efficient editing schemes, including unlearning in large language models: fine-tune on new capabilities before erasing unwanted behaviors.","The paper's evidence for hierarchy direction is visual inspection of L2 norm differences on three segmentation settings; a quantitative per-block change statistic with variance bars would turn the principle into a measurable invariant and could predict which classes are easy or hard to forget.","The boundary loss suggests a testable extension: sweeping the BND threshold and measuring the forget-versus-retain trade-off curve would reveal whether near-zero forgetting accuracy is a robust plateau or a knife-edge result."],"forward_implications":["For any two-stage knowledge-editing pipeline on a pretrained transformer, the paper predicts learn-first beats forget-first: with low-rank adaptation and group sparsity, forget-set accuracy falls to near zero while retention and learning accuracy stay high.","The reverse order should show a rebound: after an initial drop, forgetting-set accuracy climbs back because the learning phase perturbs low-level parameters that the earlier forgetting pass relied on.","Adding a second forgetting pass after forget-then-learn restores erasure, which the paper reads as evidence that the rebound is caused by low-level re-activation rather than incomplete erasure.","The ordering advantage transfers across tasks and architectures: image classification on ImageNet-100, semantic segmentation on ADE20K, and object detection on COCO all follow the same learn-first pattern."],"supporting_citations":[{"why":"Supplies the low-rank adaptation mechanism through which all learning and forgetting edits are made.","marker":"(Hu et al., 2021)"},{"why":"Supplies the claim that transformer FFN linear layers store knowledge, justifying editing those layers.","marker":"(Geva et al., 2020)"},{"why":"Defines the group lasso regularization used to zero out irrelevant LoRA groups.","marker":"(Yuan & Lin, 2006)"},{"why":"Contributes structured sparsity learning, the basis for group-wise parameter selection.","marker":"(Wen et al., 2016)"},{"why":"Introduces sparse convolutional network training that motivates the sparse constraint.","marker":"(Liu et al., 2015)"},{"why":"Provides the NegGrad+ forgetting-and-retention balancing idea that the forgetting-phase loss adapts.","marker":"(Kurmanji et al., 2024)"},{"why":"Provides the pretrained Mask2Former segmentation model used in the segmentation experiments.","marker":"(Cheng et al., 2021)"},{"why":"Provides the pretrained DINO detector used in the object detection experiments.","marker":"(Zhang et al., 2022)"}],"fun_headline_variants":["Order matters: learn before forgetting for clean model erasure","Feature hierarchy shows why learning-first beats forget-first","New swap task: learn new, keep old, forget others — in that order","Reverse forgetting: learn low-level first, erase high-level later","Knowledge swapping: sequence learning then erasure for near-zero forget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ordering argument rests on the claim that learning always builds from low-level to high-level features while forgetting erases from high-level down to low-level, a directionality the paper reads off parameter-norm plots rather than proves with quantitative layer-wise statistics.","fun_headline_variants_meta":{"raw":{"variants":["Order matters: learn before forgetting for clean model erasure","Feature hierarchy shows why learning-first beats forget-first","New swap task: learn new, keep old, forget others — in that order","Reverse forgetting: learn low-level first, erase high-level later","Knowledge swapping: sequence learning then erasure for near-zero forget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1334,"prompt_tokens":824,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":440,"tokens_out":510,"duration_ms":5084,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T10:55:38.729616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure per-block parameter-change statistics during learn-only and forget-only runs on the classification and detection models used in the paper's main tables: if learning updates are not concentrated in early blocks or forgetting updates are not concentrated in late blocks, the directional hierarchy that motivates the learn-first order is falsified.","supporting_citations":[],"review_version":1}