{"id":"b46c0fde-2cf2-4db3-a1f2-cc2744ec099b","arxiv_id":"2502.09500","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Eidetic Learning freezes each task's important neurons and prunes connections from recycled neurons, making catastrophic forgetting impossible for the retained subnetwork.","lead":"A new training method, Eidetic Learning, prevents catastrophic forgetting by pruning and freezing the neurons each task needs, then reusing the freed neurons for later tasks. The paper proves that previously learned tasks keep producing identical outputs, and it adds a learned router so no task ID is needed at inference.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The exactness proof for BN layers assumes per-channel freezing of running statistics; §3.1 does not specify how this is done, and native PyTorch BN updates all channels, so the normalization-layer guarantee is not established.","rationale":"The reader's weakest-assumption analysis identified the same load-bearing gap: the proof of exact immutability requires that every input to a frozen neuron, including BN running statistics, be held perfectly fixed, but the paper does not explain how this is achieved for a BatchNorm layer that must simultaneously serve frozen and recycled channels. This is not a question of empirical quality but of whether the stated mathematical guarantee actually covers the architectures used in the experiments. If the released code does contain a custom per-channel BN, the concern reduces to a documentation gap and the verdict stays conditional pending that clarification. If it does not, the guarantee fails for every BN-containing network, which would make the abstract's normalization-layer claim incorrect as written. I considered other potential objections, such as the unproven 'smallest set' claim and the task-classifier routing gap, but those affect optimality or practical accuracy, not the core exactness proof. The BN issue directly attacks the premise that f_theta_tj(xi) remains unchanged, so it is the most load-bearing concern. Since the reader already flagged it and assigned CONDITIONAL, my read does not change the verdict.","tokens_in":13853,"tokens_out":6452,"duration_ms":63418,"concrete_test":"Run a two-task ResNet18 (CIFAR-10) experiment using the released EideticNet code. Add a check that prints, for each frozen BN channel, (running_mean, running_var) at end of task 0 and after task 1 training, and record whether the code contains a custom BatchNorm that masks per-channel running-stat updates. Then compare task-0 test accuracy (or hidden-state distance) of (a) the released code, (b) a variant that uses native PyTorch BN with only a module-level training/eval toggle, and (c) a variant with frozen BN channels copied to separate eval-mode modules. If (a)'s running stats drift by more than machine epsilon or if (b) shows any task-0 accuracy drop, the exactness guarantee fails in the stated implementation; if (c) preserves stats exactly, the missing specification is the only issue.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Persistence and Resistance make f_theta_tj(xi) exactly equal to f_theta_ti(xi) for every previous task. For linear and convolutional layers, the implementation of Resistance is exact: zeroed input dimensions of frozen neurons are never updated. Batch normalization breaks this unless the statistics entering frozen channels are also exactly fixed. Section 3.1 says previous tasks' BN is kept in evaluation mode and running statistics are not updated, but a standard BatchNorm module has a single module-level `training` flag and updates running_mean/running_var for every channel on each training forward pass. Keeping the whole BN module in eval mode would prevent the recycled channels from learning on later tasks; keeping it in train mode updates the frozen channels' statistics. The paper does not state that the released framework implements a per-channel BatchNorm that freezes only the frozen channels' running statistics while allowing recycled channels to update theirs. Without such a mechanism, the BN output of frozen channels drifts, the hidden state changes, and the proof's exactness premise fails for the ResNet experiments. The abstract's statement that the guarantee holds for normalization layers is therefore unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Eidetic Learning, a continual-learning method that trains each task, identifies important neurons by iterative structured pruning, freezes those neurons, and prunes all synapses from pruned/recycled neurons into the frozen neurons before training the next task. The authors argue that these two operations, Persistence and Resistance, are sufficient to keep the network's output for every previous task exactly unchanged, so catastrophic forgetting is impossible by construction. The method is implemented for linear, convolutional, batch-normalization, and residual layers, and it includes a learned task classifier that routes new instances to the per-task classifier without explicit task IDs. Experiments on Permuted MNIST, Sequential CIFAR100, and Sequential Imagenette show flat test accuracy for earlier tasks and competitive per-task accuracy, with code released.","tokens_in":14069,"tokens_out":5642,"duration_ms":51931,"significance":"If the construction is implemented exactly as described, the paper offers a clean and practically useful sufficient condition for avoiding catastrophic forgetting: freezing task-important neurons and deleting incoming synapses from recycled neurons makes the representation of an earlier task invariant to later training. The presented conditions are transparent, the empirical curves are consistent with the claimed invariance, the method requires no replay, and the open-source release is a concrete strength. The paper also contributes an honest comparison against related pruning-based continual-learning methods and reports the cost of learned task routing. The main limitations are that the BatchNorm preservation mechanism is not specified at the per-channel level, which is necessary for the exactness proof in modern networks, and several claims about 'smallest' sets and end-to-end guarantees go beyond what is established.","major_comments":[{"comment":"The exactness guarantee for BatchNorm layers is not established. The text says that for previous tasks the learnable parameters remain in evaluation mode and that their internal running statistics are not updated, but a standard BatchNorm module has a single module-level training flag and updates running_mean and running_var for every channel on each training forward pass. If the whole BN module is kept in train mode, the frozen channels' statistics drift; if it is kept in eval mode, the recycled channels can no longer learn. The paper does not specify a per-channel BatchNorm implementation that freezes only the statistics of important neurons while allowing recycled channels to update. Without such a mechanism, the values of mu and sigma entering frozen channels change during later-task training, so the output of the frozen subnetwork is no longer exactly preserved and the proof's premise fails for the ResNet experiments. Please specify the exact per-channel masking procedure or restrict the formal guarantee to architectures without normalization layers.","section":"Section 3.1, Eq. (2)"},{"comment":"The algorithm is described as finding 'the smallest set of neurons necessary to perform task ti', and the introduction and Listing 1 repeat the claim that the framework selects a 'minimal' subset of neurons. However, the actual iterative-pruning procedure uses a fixed pruning step size and a stopping threshold on training-set accuracy; the paper gives no proof, and no plausible argument, that this procedure produces a minimal set of neurons. This is load-bearing for the capacity-efficiency and interpretability claims, because the method's stated benefit is that it accounts for the 'incremental amount of capacity required'. Please either provide a formal treatment of minimality or replace 'smallest' and 'minimal' with language describing an approximately-minimal set determined by the pruning hyperparameters.","section":"Section 3, Step I and Section 2"},{"comment":"The no-forgetting guarantee applies to the per-task representation under oracle task routing, while the abstract claims that an EideticNet 'automatically routes new instances without auxiliary task information' and thereby 'provably solves catastrophic forgetting'. The task classifier is a separate learned component that is not covered by the Persistence/Resistance argument, and the experiments show that it can make substantial routing errors: for the small MLP on CIFAR10-5, oracle routing achieves 80.60% on class 0 with l1 pruning while the learned router achieves 63.40%, and for ResNet50 the average per-class drop is 1.8 percentage points. The manuscript should explicitly state that the exactness guarantee is conditional on correct task routing and should specify when and how the task classifier is trained or updated across the task sequence, since an incrementally updated classifier could itself suffer from forgetting.","section":"Abstract and Section 4, Tables 7 and 8"}],"minor_comments":[{"comment":"The sentence 'we believe they can be benefit practitioners and theorists alike' contains a grammatical error ('be benefit') that should be corrected.","section":"Abstract"},{"comment":"The phrase 'seteting the targets yit of each specific task t to t' has a typo: 'seteting' should be 'setting'.","section":"Section 1"},{"comment":"The related-work section uses 'CNLP' in Table 1 but 'CLNP' in the text; the abbreviation should be consistent.","section":"Section 2"},{"comment":"The sentence 'we also to (i) make the beta, gamma of the previous tasks stay in evaluation mode...' is incomplete; the missing verb should be supplied. This is especially important because this sentence is the entire description of the BatchNorm preservation mechanism.","section":"Section 3.1"},{"comment":"The text promises 'an argument for why making self-attention layers of Transformers immune to catastrophic forgetting is challenging if not impossible', but no such argument appears in the manuscript. Either add the argument or remove the promise.","section":"Section 3.1"},{"comment":"The text contains several typos, including 'F orward transfer' with an extra space and 'Future work may extending Eidetic Learning' with an ungrammatical verb form.","section":"Section 5"},{"comment":"The appendix still contains the template sentence 'Authors may wish to optionally include extra information...' which should be removed from a submitted manuscript.","section":"Appendix, Section 6.1"},{"comment":"The table reports standard deviations for the authors' method but not for the cited baselines; please clarify whether the baseline values are taken directly from the original papers and whether the comparison is on identical train/test splits.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The central construction is simple and, modulo the BatchNorm issue, the invariance argument is sound. The main technical gap is the per-channel freezing of BN running statistics, which is essential for the claimed guarantee on modern networks and must be specified or removed from the formal claim. The 'smallest set' language is also an overclaim relative to the described heuristic pruning. The learned-task-router experiments are honest but should be framed as a limitation of the end-to-end system rather than as part of the provable guarantee. With these changes, the paper could be a solid contribution to the continual-learning literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and mostly sound: after training a task, prune unimportant neurons, freeze the important ones, and remove synapses from pruned to frozen neurons. Persistence and Resistance are sufficient conditions, and for linear/conv layers the implementation is exact. The paper honestly footnotes that the sparsity pattern is the same as Golkar et al. 2019, so what is genuinely new is the adaptive accuracy-based pruning, the explicit handling of normalization and residual layers, and the learned task router for inference without task IDs. That is a real but incremental contribution.\n\nThe main soft spot is batch normalization. The paper claims the guarantee holds for normalization layers, but Section 3.1 only says previous tasks' BN is kept in evaluation mode and running statistics are not updated. A standard PyTorch BN module updates all channels' running statistics on every training forward pass. Without a per-channel freezing mechanism keeping only frozen channels' statistics fixed while recycled channels still learn, the exactness guarantee fails for ResNets. This is fixable in implementation but must be specified; as written, the BN claim is unsupported. The stress-test note lands on this accurately.\n\nThe other issues are less severe. The abstract's \"provably solves catastrophic forgetting\" overclaims: the paper only addresses task-incremental learning and requires excess capacity, both of which the conclusion admits. Calling N_ti the \"smallest set\" of neurons is not justified; the procedure just prunes until accuracy drops. Deep-network experiments in Tables 4 and 5 have no baselines, so it is hard to know how competitive the method is beyond Permuted MNIST. The learned task router is a separate supervised component, and the reported accuracy drops (e.g., 1.8% on CIFAR-100) mean the \"no task IDs\" claim is only as strong as the router.\n\nThat said, the central mechanism is coherent, the empirical retention curves are flat as expected, and the code is released. This is a legitimate baseline for task-incremental continual learning, not a revolution. With a clarified BN implementation, added baselines, and a more precise abstract, it would be a solid paper. Send it to peer review, but the reviewers should push on those details.","headline":"A solid task-incremental continual learning method built on prune-and-freeze, with a sound core proof for linear and convolutional layers; the BN exactness claim is under-specified and the abstract overreaches, but the mechanism deserves serious engagement.","tokens_in":14601,"tokens_out":1975,"would_cite":true,"duration_ms":19071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A network trained with Eidetic Learning cannot forget: freezing a task's important neurons and deleting synapses from recycled neurons into them makes its output on that task exactly immutable.","keywords":["catastrophic forgetting","continual learning","iterative pruning","neuron freezing","task-incremental learning","mixture of experts","batch normalization"],"falsifier":"On a model with batch normalization, train an EideticNet on task 1, freeze its important neurons and their batch-normalization running statistics, then train task 2 in ordinary training mode; recompute task 1's outputs. If the frozen channels' running mean or variance moved, or if any task-1 test input has a different output before and after task 2, exact immutability is refuted for modern normalization-layer networks.","tokens_in":13640,"feed_emoji":"🧠","tokens_out":6346,"duration_ms":50727,"temperature":0.7,"pith_summary":"This paper sets out to make catastrophic forgetting impossible by construction rather than less likely. Its proposed training procedure, Eidetic Learning, trains each task, freezes the neurons that task needs, deletes every synapse leading from the remaining 'recycled' neurons into those frozen neurons, and then reuses the recycled neurons for the next task. The claim is that any network trained this way—an EideticNet—produces exactly the same outputs on every previously learned task after later tasks are trained, with no rehearsal or replay. If true, continual learning in overparameterized networks becomes a capacity-budgeting problem: as long as each new task fits in the spare neurons, old skills are retained exactly. The paper also shows how a learned task classifier lets the network route new examples without being told which task they come from.","feed_headline":"Freeze key neurons, wipe the rest: forgetting becomes impossible","feed_subtitle":"Freezing task-critical neurons and severing their incoming synapses makes every learned task exactly unchangeable.","key_machinery":"The object that carries the argument is the pair of conditions (I) Persistence and (II) Resistance, stated in Section 3 as sufficient for exact retention of $f_{\\theta_{t_j}}(x_i)$. The implementing mechanism is iterative structured pruning with a nested sparsity pattern: a per-task neuron mask selects the minimal set $N_{t_i}$; frozen neurons $F$ accumulate; neurons in $R = N \\setminus F$ are reinitialized and their synapses into $F$ are permanently zeroed. For batch-normalization layers, the learned scale and shift are pruned with the layer mask, and the running statistics of frozen channels must be kept in evaluation mode and not updated during later tasks.","core_discovery":"The paper's central claim is that two conditions—persistence (neurons important to task $t_i$ are unchanged during later tasks) and resistance (no signal flows from non-important neurons into those important neurons)—are together sufficient to make $f_{\\theta_{t_j}}(x_i)$ identical to $f_{\\theta_{t_i}}(x_i)$ for every input $x_i$ and every later task $t_j$. Eidetic Learning implements these conditions by iterative structured pruning: train a task, prune the smallest set of neurons whose removal keeps training accuracy above a threshold, freeze the survivors, delete the synapses from pruned to frozen neurons, reinitialize the pruned neurons, and train the next task. The unusual architectural choice is nesting: later tasks can use earlier tasks' frozen features through remaining synapses, while the deleted synapses guarantee that recycled neurons cannot affect frozen ones. If the implementation really holds batch-normalization running statistics fixed per frozen channel, no later training step can alter any earlier task's function.","pith_inferences":["Editorial inference: if exact immutability holds, catastrophic forgetting is reframed as a capacity-allocation problem: an EideticNet is correct whenever each new task fits in the remaining recycled capacity, making 'excess capacity' a measurable resource.","Editorial inference: the method's explicit assumption of task-incremental learning suggests a natural test—overlapping classes would require a different routing or multi-head arbitration mechanism, since the learned task classifier is supervised.","Editorial inference: the backward-transfer extension described in the paper would let later-learned features improve earlier tasks, but it requires re-training the earlier task on its original data when the data distribution is non-stationary, which is an implicit replay requirement."],"forward_implications":["If the two conditions are met, the network's output on every past task is immutable, so forgetting is impossible by construction rather than reduced in probability.","Because important neurons are frozen and incoming synapses from recycled neurons are deleted, later tasks can reuse early-task features without disturbing them; only forward transfer is supported.","The method needs no rehearsal or replay and adds only a constant number of hyperparameters per network, with time and space complexity linear in parameter count.","A learned task classifier on the penultimate hidden states removes the need for a task ID at inference, at some measured cost in per-class accuracy in the reported CIFAR-100 experiments.","The same guarantee extends to convolutional, residual, and normalization layers, and to fine-tuning after pre-training, as long as frozen batch-normalization statistics are held fixed."],"supporting_citations":[{"why":"Defines the catastrophic-interference problem that the paper's guarantee is designed to eliminate.","marker":"[McCloskey and Cohen, 1989]"},{"why":"Supplies the iterative pruning procedure used to find the minimal neuron set for each task.","marker":"[Han et al., 2015]"},{"why":"Introduced the nested sparsity pattern that EideticNets use so later tasks reuse earlier features.","marker":"[Golkar et al., 2019]"},{"why":"Provides the Taylor pruning score used as one of the neuron-importance criteria.","marker":"[Molchanov et al., 2017]"},{"why":"Defines batch normalization, whose per-channel running statistics the proof must hold fixed to preserve frozen-task outputs.","marker":"[Ioffe and Szegedy, 2015]"}],"fun_headline_variants":["Eidetic Learning: provable end to catastrophic forgetting","Freeze key neurons, sever synapses, never forget","EideticNets: linear complexity, zero forgetting","Task freezing makes neural forgetting provably impossible","Catastrophic forgetting: Eidetic Learning's provable fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of exact retention assumes that a frozen channel's batch-normalization running statistics can be held bit-for-bit at their end-of-task values while other channels train in standard mode, but standard frameworks update all channels' statistics during training and the paper specifies no per-channel freezing mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Eidetic Learning: provable end to catastrophic forgetting","Freeze key neurons, sever synapses, never forget","EideticNets: linear complexity, zero forgetting","Task freezing makes neural forgetting provably impossible","Catastrophic forgetting: Eidetic Learning's provable fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1594,"prompt_tokens":999,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":517}},"tokens_in":615,"tokens_out":595,"duration_ms":6693,"temperature":1.0,"reasoning_tokens":517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:15:23.130619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a model with batch normalization, train an EideticNet on task 1, freeze its important neurons and their batch-normalization running statistics, then train task 2 in ordinary training mode; recompute task 1's outputs. If the frozen channels' running mean or variance moved, or if any task-1 test input has a different output before and after task 2, exact immutability is refuted for modern normalization-layer networks.","supporting_citations":[{"cited_title":"Catastrophic interference in connectionist networks: The sequential learning problem","cited_arxiv_id":null,"evidence_quote":"Defines the catastrophic-interference problem that the paper's guarantee is designed to eliminate."},{"cited_title":"Pruning convolutional neural networks for resource efficient inference","cited_arxiv_id":null,"evidence_quote":"Provides the Taylor pruning score used as one of the neuron-importance criteria."}],"review_version":1}