{"id":"acbe7a3f-0cd2-4c42-b5e7-4c545f1d97a9","arxiv_id":"2608.12874","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A learnable wavelet activation with dynamic capacity injection and slope regularization improves plasticity retention in continual learning.","lead":"This paper introduces a learnable activation function that combines a stable smooth base with adjustable wavelet bumps, aiming to prevent continual learning models from losing the ability to learn new tasks. It also adds a mechanism that injects extra wavelets when progress stalls, plus a regularizer that protects old knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed gains may come from the decoupled learning-rate scheme, not the wavelet activation; baselines are not tested under the same optimization protocol.","rationale":"I focused on the empirical confound rather than Assumption 1 because the headline claim is empirical superiority. If the decoupled LR is the cause, the method's main contribution is undermined; the theoretical caveats would be secondary. The reader's weakest_assumption (Assumption 1) is a real weakness in the 'rigorous guarantees' section, but a circular assumption does not make the experiments invalid, whereas an uncontrolled optimization variable does. The paper's ablations isolate the activation components, but they do not isolate the optimization scheme from the activation when comparing to baselines. Therefore, the most load-bearing concern is the missing control for the decoupled learning rate. A single experiment applying the same LR protocol to standard activations would settle it. The reader's verdict of CONDITIONAL is appropriate; the concern adds a specific condition rather than overturning the overall assessment.","tokens_in":18904,"tokens_out":9709,"duration_ms":99767,"concrete_test":"Re-run the CIFAR-100 experiments in the SSD and EWC/WA settings with ReLU, AID, and Randomized Smooth-Leaky activations, applying exactly the same decoupled-LR protocol used for ChannelWavAct—λ_lr multiplier on backbone and activation parameters, η_base on biases, and optimizer rebuild at the corresponding injection iterations—while keeping the activation fixed. If any of these baselines reaches or exceeds ChannelWavAct's reported accuracy, the performance gains are attributable to the optimization scheme rather than the wavelet activation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ChannelWavAct 'consistently outperforms existing baseline activation functions' (Section I-B). The full method couples the wavelet activation with a decoupled learning rate: backbone weights and activation parameters use η_high = λ_lr·η_base, while biases/BatchNorm use η_base (§III-B3). In all comparisons (Tables I and II), the baselines (ReLU, AID, Randomized Smooth-Leaky, Rational, B-Spline) are trained with the standard uniform LR. The ablation in Table IV shows that this decoupled LR is worth ~4.7% Average Accuracy and 7.0% reduction in Forgetting for ChannelWavAct (rows #3 vs #4). Yet the same decoupled-LR protocol is never applied to the baselines. A higher LR on backbone weights is a generic plasticity intervention; if it alone closes the gap, the activation design is not what drives the reported SOTA results. The optimizer reset during injection is a second generic confound not applied to baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ChannelWavAct, a channel-wise learnable wavelet activation for continual learning. The activation is a sum of a fixed SiLU base for low-frequency content and learnable Mexican-hat wavelets for high-frequency detail, with dynamic wavelet injection triggered by loss stagnation, slope-specific regularization, and a decoupled learning-rate schedule. The paper claims rigorous theoretical guarantees: that the hybrid wavelet architecture is necessary for efficient L2 approximation (Theorem 4), that the decoupled learning-rate mechanism restores plasticity for high-frequency information (Theorem 2), and that the loss-driven trigger is formally justified (Theorem 3). It reports extensive experiments on Permuted MNIST, Random Label MNIST, CIFAR-100, Mini-ImageNet, Tiny-ImageNet, and ImageNet-100 in both replay-based and replay-free continual learning settings, plus spectral analysis experiments.","tokens_in":19107,"tokens_out":8561,"duration_ms":93298,"significance":"If the results hold, the paper offers a useful and concrete activation-level intervention for plasticity loss, with the appealing idea of explicitly separating low-frequency stability from high-frequency adaptability. The empirical scope is broad, including multiple benchmarks, two continual learning frameworks, ablations, parameter sensitivity, and spectral diagnostics. The theoretical sections are standard in flavor but do not currently provide the promised 'rigorous mathematical guarantees': Theorem 2 depends on an assumption that is essentially the property being argued for, and Theorem 3 is not connected to the actual trigger hyperparameters. The main empirical comparison also needs a control for the decoupled optimization protocol before the performance claim can be attributed to the wavelet activation. These are fixable concerns, so the paper warrants a major revision rather than rejection.","major_comments":[{"comment":"Theorem 2 assumes that the wavelet kernel already dominates the high-frequency direction, with Θ_high(t)v_high = λ_high(t)v_high and negligible leakage ε(t). The theorem then 'proves' that a decoupled learning rate amplifies the decay of high-frequency residuals. But the assumption is precisely the substantive property that the wavelet activation is claimed to provide, so the result does not establish that ChannelWavAct achieves this alignment or that the decoupled rate is necessary. Relatedly, Theorem 1 invokes the Frequency Principle to assert λ_high ≪ λ_low, which is a background empirical observation, not a theorem. The claims in Section IV that 'the results of Theorem 2 demonstrate that our decoupled learning rate mechanism effectively compensates for the deficiency' should be substantially softened, or the paper should provide empirical evidence (e.g., measuring the projection of task residuals onto the learned wavelet span during training) that the assumed alignment actually occurs.","section":"Section IV, Assumption 1 and Theorem 2 (Eqs. 16, 19, 20)"},{"comment":"The main cross-method comparisons train ChannelWavAct with the decoupled learning rate (η_high = λ_lr·η_base on backbone weights and activation parameters) and with optimizer rebuilding at injection, but the baseline activations are not documented as receiving the identical optimization protocol. Table IV in Section V-C shows that the decoupled rate alone contributes 4.7% average accuracy and 7.0% forgetting reduction for ChannelWavAct, and the same table suggests ReLU also improves by about 5.5 points when the decoupled protocol is applied. Unless all baselines in Tables I and II are rerun with the same decoupled-learning-rate and optimizer-reset protocol, the statement that 'our approach consistently outperforms existing baseline activation functions' over-attributes the gains to the wavelet activation rather than to a generic plasticity-enhancing optimization intervention. The paper should report baseline results under the identical optimization protocol in the main tables.","section":"Section V, Tables I, II and Table IV"},{"comment":"Theorem 3 derives a minimum injection size Δ* = ⌈μ(Ω_res)/B_ψ⌉ in Eq. (22) and claims to justify the loss-stagnation trigger. However, the trigger mechanism in Eq. (6) uses a counter with patience P and margin δ, and no formal connection is made between P, δ, and the Rayleigh-quotient threshold ε_tol from Eq. (21). The sensitivity analysis in Fig. 8 finds Δ = 1, P = 50, δ = 0.005 optimal, and the paper does not use Δ* to set any hyperparameter. The claim of providing 'a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection' is therefore not supported by the presented theory. The authors should either derive a practical calibration of P and δ from the theorem or rephrase the contribution.","section":"Section IV, Theorem 3 and Section V-C, Fig. 8"}],"minor_comments":[{"comment":"The citation markers '[cite: 1063, 1064]' appear unresolved in the text and should be replaced with proper references.","section":"Section V-A, paragraph on Random Label MNIST"},{"comment":"The asterisk on 'ChannelWavAct*' is not defined in the caption or body text; the distinction between ChannelWavAct* and ChannelWavAct, and the relation to 'w/o L2 Regularization' and '+L2 Regularization', should be clarified.","section":"Table II"},{"comment":"The notation σ is inconsistent: Eq. (5) defines σ(x) = SiLU(x), while Theorem 4 writes φ(x) = x·σ(x) and calls this the SiLU-based scaling function; the reader is left to infer that σ is the sigmoid in one place and SiLU in another. Please use distinct symbols or state the convention explicitly.","section":"Section III-A, Eq. (5), and Section IV, Theorem 4"},{"comment":"The algorithm resets L*_min and C_bad at the start of each epoch, so the stagnation trigger only monitors within a single epoch and cannot detect cross-epoch plateaus. This is not obviously the same as the loss-stagnation notion analyzed in Theorem 3, and the discrepancy should be discussed.","section":"Algorithm 2"},{"comment":"The paper describes the results as 'state-of-the-art,' but the comparisons are limited to activation-function baselines within fixed frameworks (SSD, EWC, W A). Recent plasticity-loss methods that act through weight re-initialization or regularization (e.g., Shrink and Perturb, regenerative regularization) are cited in the related work but not compared against; the authors should either include such comparisons or qualify the claim.","section":"Section I-B and V"}],"recommendation":"major_revision","confidential_remarks":"I do not see a fatal flaw in the core empirical idea; the reported gains are consistent across many settings, and the ablation shows that the decoupled learning rate is important but not the whole story. The main work needed before publication is to (1) rerun baselines under the same optimization protocol, (2) reframe the theoretical claims to match what is actually proven, and (3) either connect the injection-trigger theory to the implemented mechanism or remove the 'formal derivation' claim. The paper is within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the empirical core is worth a look, but the theory doesn't do the load-bearing work the paper claims, and the optimization protocol is confounded with the activation change. The reported gains are probably real, but the size is overstated until baselines are trained under the same decoupled-LR scheme.\n\nWhat's genuinely new: a channel-wise activation that sums a SiLU base with learnable Mexican Hat wavelets, plus dynamic injection of new wavelets on a loss-stagnation trigger and slope-specific regularization that freezes old weights while letting translations and scales adapt. The spectral-analysis experiments are a nice addition: they demonstrate the mechanism does learn high frequencies faster than ReLU. Ablations show each component contributes something; Table III gives a clean decomposition.\n\nThe soft spots are where the stress-test note lands. The full method always couples the wavelet activation with a decoupled learning rate (higher on backbone weights and activation parameters, base on biases/BN). Baselines in Tables I and II are trained with a uniform LR. Table IV says the decoupled LR alone is worth about 4.7% Avg and 7% Fgt for ChannelWavAct, so it is not a small detail. The paper's central claim—'consistently outperforms existing baseline activation functions'—needs baselines under the same optimization protocol before it can be taken at face value. A generic higher LR on backbone weights is a known plasticity intervention; the unique contribution of the wavelet design is left unidentified.\n\nThe theory section is weaker than the abstract implies. Assumption 1 states that wavelet bases dominate the high-frequency subspace and the base leakage is negligible; Theorem 2 then proves the decoupled LR accelerates the decay of high-frequency residuals. That is the conclusion already packed into the assumption. Theorem 3's loss-stagnation characterization is a restatement of the Rayleigh quotient collapse, and Theorem 4 is a standard zero-mean wavelet approximation fact dressed up as 'structural necessity.' None of this invalidates the method, but 'rigorous mathematical guarantees' overstates what is actually proven. Minor issues: no code release, no variance in Table II, and hyperparameters are tuned on the same benchmarks used for the headline numbers.\n\nFor whom: researchers working on activation design for continual learning, and anyone evaluating the plasticity-loss literature. It deserves a serious referee, but with a request for revised claims and additional experiments under matched optimization. I'd send it to review.","headline":"Promising empirical activation for continual learning, but the theory overreaches and the decoupled-LR confound inflates the headline gains.","tokens_in":19621,"tokens_out":2408,"would_cite":false,"duration_ms":25414,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a learnable wavelet activation that splits each channel into a low-frequency base and high-frequency wavelets sustains plasticity in continual learning and outperforms both fixed and fully learnable activations.","keywords":["continual learning","plasticity loss","learnable activation","wavelet activation","spectral bias","decoupled learning rate","class-incremental learning","catastrophic forgetting"],"falsifier":"Measure the preconditioned NTK of a continual-learning ResNet using ChannelWavAct after several tasks, project the residual onto high-frequency eigendirections, and check whether $\\eta_{high}\\lambda_{high}(t)+\\eta_{base}\\epsilon(t)$ actually controls the decay rate. If residual decay stays slow even with a large $\\eta_{high}$, or if the wavelet basis fails to align with the directions the new task needs ($\\lambda_{high}$ near zero), Theorem 2's mechanism is not what drives the gains; alternatively, replacing the Mexican Hat wavelets with random high-frequency bumps of the same scale should leave performance nearly unchanged if only the learning rate matters.","tokens_in":18678,"feed_emoji":"🧠","tokens_out":5375,"duration_ms":51281,"temperature":0.7,"pith_summary":"This paper tries to establish that the activation function itself is a major, underused lever for preventing plasticity loss in continual learning. It proposes ChannelWavAct, which writes each channel's activation as a fixed low-frequency base (SiLU) plus a sum of learnable high-frequency Mexican Hat wavelets, and adds a loss-stagnation trigger that injects new wavelets when learning plateaus. The paper argues that this hybrid structure counters the spectral bias of fixed activations while regularization on wavelet amplitudes keeps earlier tasks from being overwritten. If correct, it gives a drop-in replacement for ReLU in continual learning models that sustains trainability over hundreds of tasks without replay buffers.","feed_headline":"Wavelet activations keep neural nets plastic across 200 tasks","feed_subtitle":"A hybrid low/high-frequency activation with adaptive capacity injection beats fixed and fully learnable baselines.","key_machinery":"The central object is ChannelWavAct, a channel-wise activation $\\Phi_c(x_c)=w_{low,c}\\sigma(x_c)+\\sum_{k=1}^{K} w_{c,k}\\psi((x_c-\\tau_{c,k})/s_k)$, with $\\sigma$ the SiLU base for low frequencies and $\\psi(u)=(u^2-1)e^{-u^2/2}$ the Mexican Hat mother wavelet for high frequencies. Three mechanisms carry the argument: (1) dynamic wavelet injection, triggered when the training loss fails to drop by a relative margin for $P$ iterations, adds new wavelets with zero-initialized weights and translations so capacity grows without perturbing the learned state; (2) slope-specific regularization penalizes only old amplitude weights $w$, leaving translations and scales free; (3) a decoupled optimizer gives backbone and activation parameters a larger learning rate $\\eta_{high}=\\lambda_{lr}\\eta_{base}$ than biases. The theoretical work uses the local dynamic NTK and a frequency-domain covering argument to show why each of these pieces is needed.","core_discovery":"The central claim is that decomposing the activation into a global low-frequency part and localized high-frequency wavelets is both necessary and sufficient to sustain plasticity: the pure-wavelet part cannot approximate non-zero-mean signals without divergent scales, so the low-frequency base is structurally required, and the wavelet part is what lets high-frequency residuals decay at a useful rate. The paper formalizes this with an NTK analysis: under a uniform learning rate, high-frequency eigenvalues are too small to shrink residuals, but a decoupled learning rate that scales up backbone and activation parameters multiplies the high-frequency eigenvalue and restores convergence. It also proves that loss stagnation coincides with the residual entering the approximate null space of the kernel, which justifies the injection trigger, and gives a spectral-covering formula for how many new wavelets to add. Empirically, replacing ReLU with ChannelWavAct in ResNet backbones improves average and last accuracy on class-incremental benchmarks while keeping forgetting comparable.","pith_inferences":["A natural testable extension is to apply ChannelWavAct to deep reinforcement learning, where plasticity loss and dormant neurons are measured directly; the dormant-ratio results in the paper suggest the same mechanism should keep units alive there.","The loss-stagnation trigger could be replaced by a rank-based or NTK-based detector; if the Rayleigh-quotient characterization in Theorem 3 is right, any detector of residual null-space entry should give similar injection timing.","The spectral robustness result implies that high-frequency components are structurally anchored in dedicated bases, which may also help in fine-tuning or sparse-replay regimes where parameter drift is common.","If the computational overhead the paper acknowledges can be reduced, this activation could become a default choice for long-horizon learning, not just class-incremental benchmarks."],"forward_implications":["In class-incremental settings, swapping ReLU for ChannelWavAct should raise average accuracy by two to three points on replay-based benchmarks and by larger margins in replay-free settings, according to the reported results.","The decoupled learning rate is not a tuning convenience but a necessary condition for high-frequency learning under the paper's analysis; without it the wavelet bases cannot converge fast enough.","The loss-stagnation trigger gives a practical, parameter-free way to detect capacity saturation: no separate plasticity metric or gradient statistics are required.","A hybrid base-plus-wavelet form is required for $L^2$ approximation of non-zero-mean signals; pure wavelet activations would need divergent scales and become unstable in finite networks.","Because post-activation batch normalization is part of the module, injected wavelets can be added without destabilizing feature distributions, which supports long sequences of injections."],"supporting_citations":[{"why":"Defines the Permuted MNIST and Random Label MNIST plasticity-loss protocol and the dormant-neuron framing the paper builds on.","marker":"[17]"},{"why":"Supplies the Frequency Principle claim that high-frequency NTK eigenvalues decay much slower than low-frequency ones, the basis for Theorems 1 and 2.","marker":"[36]"},{"why":"Provides the local dynamic NTK formalism used to model residual evolution and motivate the decoupled learning rate.","marker":"[35]"},{"why":"Serves as the main static stochastic activation baseline that ChannelWavAct must beat on trainability benchmarks.","marker":"[25]"},{"why":"Represents the learnable B-spline activation baseline whose unrestricted updates cause forgetting and whose scalability the paper contrasts with wavelets.","marker":"[31]"},{"why":"Establishes spectral bias of standard networks, the phenomenon the wavelet decomposition is designed to counter.","marker":"[28]"},{"why":"Defines effective rank, the representation-quality metric used to show ChannelWavAct keeps a rich feature space.","marker":"[15]"},{"why":"Provides the replay-based continual learning framework used for the generalizability comparison tables.","marker":"[4]"}],"fun_headline_variants":["Learnable wavelets prevent plasticity loss in sequential learning","Hybrid wavelet activations: plasticity without catastrophic forgetting","Wavelet activations counter spectral bias to keep continual learning plastic","Wavelets break spectral bias to sustain long-run continual learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that decoupled learning rates restore high-frequency plasticity rests on Assumption 1: high-frequency features are nearly orthogonal to the low-frequency base and the injected wavelets genuinely dominate the high-frequency subspace, so the base contributes only negligible leakage to those directions.","fun_headline_variants_meta":{"raw":{"variants":["Learnable wavelets prevent plasticity loss in sequential learning","Hybrid wavelet activations: plasticity without catastrophic forgetting","Wavelet activations counter spectral bias to keep continual learning plastic","Wavelets break spectral bias to sustain long-run continual learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001152,"raw_usage":{"total_tokens":4766,"prompt_tokens":929,"completion_tokens":3837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3771}},"tokens_in":545,"tokens_out":3837,"duration_ms":26367,"temperature":1.0,"reasoning_tokens":3771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:30:53.632481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the preconditioned NTK of a continual-learning ResNet using ChannelWavAct after several tasks, project the residual onto high-frequency eigendirections, and check whether $\\eta_{high}\\lambda_{high}(t)+\\eta_{base}\\epsilon(t)$ actually controls the decay rate. If residual decay stays slow even with a large $\\eta_{high}$, or if the wavelet basis fails to align with the directions the new task needs ($\\lambda_{high}$ near zero), Theorem 2's mechanism is not what drives the gains; alternatively, replacing the Mexican Hat wavelets with random high-frequency bumps of the same scale should leave performance nearly unchanged if only the learning rate matters.","supporting_citations":[{"cited_title":"Loss of plasticity in deep continual learning,","cited_arxiv_id":null,"evidence_quote":"Defines the Permuted MNIST and Random Label MNIST plasticity-loss protocol and the dormant-neuron framing the paper builds on."},{"cited_title":"Neural tangent kernel: Con- vergence and generalization in neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the local dynamic NTK formalism used to model residual evolution and motivate the decoupled learning rate."},{"cited_title":"Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss","cited_arxiv_id":"2502.01342","evidence_quote":"Serves as the main static stochastic activation baseline that ChannelWavAct must beat on trainability benchmarks."},{"cited_title":"Summarizing stream data for memory-constrained online continual learning,","cited_arxiv_id":null,"evidence_quote":"Provides the replay-based continual learning framework used for the generalizability comparison tables."}],"review_version":1}