{"id":"401ddb89-354b-4ffa-8262-fb46a32c0028","arxiv_id":"2501.10861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A continual learning method that uses moment-propagation uncertainty to identify important network parameters and regularizes them either by lowering their learning rates or by increasing per-parameter KL penalties.","lead":"This paper proposes two ways to stop neural networks from forgetting old tasks when learning new ones: giving important parameters smaller learning rates, and more strongly pulling them back toward their previous values. The methods use a Bayesian-style uncertainty estimate to decide which parameters matter, and the authors report improved task-incremental learning scores on several image benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing concern is the incorrect KL term in Eq. 6: since LRA/PPBI derive importance from a variational posterior, optimizing the stated ELBO with a wrongly formed KL means the learned uncertainty is not the claimed Bayesian quantity.","rationale":"The reader's conditional verdict is appropriate and remains the right disposition. The reader's weakest assumption—that the moment-propagation uncertainty estimate is accurate and transferable across tasks—is closely related to the concern I identify, but my stress-test sharpens it to a specific, checkable defect: Eq. 6 is not the Gaussian KL used to justify the variational objective. This is not merely a notational slip; it changes the training objective for every parameter, and since importance is defined from learned posterior statistics, the method's central mechanism is not the Bayesian one described. The reader also correctly notes the absence of error bars or seeds and the grid search over all tasks; those issues compound the theoretical one, because without seeded reruns one cannot tell whether small differences in Table I are signal or noise. I also credit the paper's honest reporting: the authors acknowledge marginal gains over Feature Freezing and that HAT outperforms their methods on the hardest benchmark, which tempers the abstract's claim of broad superiority. Nevertheless, the empirical core could in principle survive the KL correction, so the right verdict remains CONDITIONAL rather than REJECT: the paper should be accepted only if the authors correct Eq. 6 and show the reported results are stable under that correction and across seeds.","tokens_in":12179,"tokens_out":6502,"duration_ms":76059,"concrete_test":"Re-run the 5-Split MNIST and 2-Split CIFAR10 experiments with Eq. 6 replaced by the exact per-parameter KL, 1/2[(μq−μp)²/σp² + σq²/σp² −1 + ln(σp²/σq²)], keeping the same hyperparameter grid, architectures, and at least 5 seeds per condition. If ACC/BWT move by more than about one point, the reported improvements are tied to the misspecified KL and the central claim is not supported as stated; if the numbers are essentially unchanged, the empirical conclusion is robust despite the derivation error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MP-learned parameter uncertainty identifies important parameters and regularizes task-incremental learning—depends on the posterior being optimized via the stated ELBO in Eq. 1. Section II-A's Eq. 6 is not the Gaussian KL required by Eq. 1. For q(ω)=N(μq,σq²) and p(ω)=N(μp,σp²), the exact per-parameter KL is 1/2[(μq−μp)²/σp² + σq²/σp² −1 + ln(σp²/σq²)]. Equation 6 instead contains (μq−μp)²/σq², dividing the mean-difference term by the current posterior variance rather than the prior variance. With the stated sparsity prior σp²=1, this replacement penalizes low-variance (high-confidence) parameters far more heavily than the true KL, coupling the mean penalty to the variance in a way that is not part of the variational objective. Consequently, the parameter uncertainties and importance scores used by LRA and PPBI reflect a misspecified regularizer, not the claimed Bayesian posterior. This matters for the empirical claim because importance is never independently validated across tasks: the pruning evidence in Section III is single-task, and Table I reports no seeds or error bars. The observed CL gains could therefore be artifacts of an ad hoc objective rather than evidence that moment-propagation uncertainty gives a transferable importance measure. I am not claiming fraud or that the method must fail; the concern is that the paper's stated theoretical foundation is internally inconsistent, and the empirical design does not isolate this error.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes two continual learning regularizers, Learning Rate Adaptation (LRA) and Per-Parameter Bayesian Inference (PPBI), built on a Moment Propagation (MP) variational framework. The MP network learns per-parameter means and variances and propagates the first two moments through the network, allowing the ELBO to be evaluated without sampling. Parameter variance or SNR is then used as an importance measure to set per-parameter learning rates (LRA) or per-parameter KL regularization weights (PPBI) when training subsequent tasks. The authors evaluate on seven task-incremental benchmarks, reporting Average Test Accuracy and Backward Transfer, and compare against EWC, SI, MAS, UCL, UCB, and HAT, concluding that their methods outperform sampling-based uncertainty methods and are competitive with HAT.","tokens_in":12493,"tokens_out":5938,"duration_ms":63646,"significance":"If the uncertainty estimates are correct, the paper would make a useful contribution: it offers a sampling-free Bayesian route to continual learning and reports competitive or better ACC/BWT than the UCL/UCB baselines and several standard regularizers. The explicit comparison with the Feature Freezing baseline is honest and informative, and the pruning analysis in Section III is a sensible sanity check for the importance measure. However, the contribution is conditional: the incorrect KL term in Eq. (6) means the learned uncertainties do not correspond to the claimed variational posterior, and the absence of seeds, error bars, and cross-task importance validation limits the support for the central transferability claim.","major_comments":[{"comment":"The KL term in Eq. (6) is not the Gaussian KL required by Eq. (1). For q(ω)=N(μ_q,σ_q²) and p(ω)=N(μ_p,σ_p²), the per-parameter KL is 1/2[(μ_q−μ_p)²/σ_p² + σ_q²/σ_p² − 1 + ln(σ_p²/σ_q²)]. Equation (6) instead divides the mean-difference term by σ_q²; with the stated prior σ_p=1, μ_p=0, this penalizes low-variance parameters far more heavily than the true KL. Because the paper's importance measures are read directly from the optimized posterior variances and SNRs, the learned uncertainties do not correspond to the claimed Bayesian posterior, and the central regularizer is misspecified. The authors should correct the formula and re-run the experiments, or alternatively justify the modified objective as a deliberate regularizer and validate the importance measure independently of the ELBO derivation.","section":"II-A, Eq. (6)"},{"comment":"The claim that 'the maximum variance of a parameter present in the network will be σ²=1' is not a consequence of the sparsity-inducing prior. In variational inference, the ELBO trades off the KL term against the expected log-likelihood; if the data likelihood is improved by predictive variance, individual posterior variances can exceed the prior variance. The subsequent pruning analysis in Section III relies on this bound to interpret variance as importance, so this assertion needs a proof or a corrected, empirically supported statement.","section":"II-B"},{"comment":"The empirical central claim is not supported with uncertainty quantification. Table I reports a single run per method and no seeds, yet many comparisons are within a few tenths of a percent (e.g., 99.40% vs 99.37% for the two PPBI variants). Moreover, the pruning experiments in Section III are performed on a single trained network and do not test whether importance scores computed after task t predict damage on task t when training on task t+1; this transferability is exactly the load-bearing property for LRA and PPBI. Please provide multiple seeds with error bars and a direct cross-task validation of the importance measure, otherwise the observed gains may reflect hyperparameter selection rather than a transferable uncertainty signal.","section":"Table I and Section III"}],"minor_comments":[{"comment":"The text below Eq. (7) says the learning-rate range is defined by 'αmin and αmin'; the second occurrence should presumably be αmax.","section":"IV-A, Eq. (7)"},{"comment":"The conclusion contains a typo: 'both mitigate catastrophic forgetting through regulariz the learning' should read 'through regularizing the learning'.","section":"VI"},{"comment":"The definition SNR = |μ|/σ² is unconventional; SNR is usually defined with the standard deviation, not the variance, in the denominator. Please clarify why the variance is used and whether this choice affects comparability with UCB/UCL.","section":"III"},{"comment":"Several references are incomplete, missing volume, page, or venue information (e.g., [1], [2], [3], [6], [7], [8]); these should be completed for a journal submission.","section":"References"},{"comment":"The experimental setup does not clearly state whether task identity is provided during inference for the multi-head network on each benchmark; please specify the task-incremental protocol for all compared methods.","section":"IV-C"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the Eq. (6) issue is a concrete, checkable error, and since all experiments are trained under the stated ELBO, the empirical tables cannot be taken as evidence for the Bayesian interpretation as written. I still view this as major revision rather than rejection: the method may perform similarly with a corrected KL term, and the empirical design could be repaired with seeds and a cross-task importance validation. I would also weigh the absence of code and data in the revision decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you'll want to know about: Angelini and Bouaynaya apply their Moment Propagation (MP) framework to continual learning, replacing sampling-based uncertainty estimates (BBB) with moment propagation for two regularization schemes: LRA (learning-rate adaptation) and PPBI (per-parameter Bayesian inference). The empirical work is real: seven benchmarks, two network architectures, re-implemented baselines, and an honest admission that feature freezing is a strong competitor. They also show MP-based pruning identifies important parameters in a single-task setting. That part is decent.\n\nThe soft spot: the variational derivation has a load-bearing error. Equation (6) writes the per-parameter KL as proportional to -1 + (μq-μp)^2/σq^2 + ln(σp^2/σq^2) + σq^2/σp^2. The Gaussian KL is (1/2)[(μq-μp)^2/σp^2 + σq^2/σp^2 - 1 + ln(σp^2/σq^2)]. The mean-difference term is divided by the posterior variance σq^2 instead of the prior variance σp^2. Since they use a standard normal prior (σp^2=1), this replacement changes the objective: low-variance (high-confidence) parameters get a much larger mean penalty. So the learned uncertainties are not the claimed Bayesian posterior quantities, and the importance scores derived from them reflect a modified objective. That is not a cosmetic typo; the central claim that MP uncertainty \"identifies important parameters\" under a variational posterior is compromised.\n\nThere are other concerns: no error bars or seeds anywhere, and the hyperparameters are grid-searched to maximize performance across all tasks. That selection pressure is real, though it's common in this literature. The pruning experiment is single-task, so we don't see whether importance transfers across tasks.\n\nWhat's genuinely useful: the paper shows MP can be plugged into existing uncertainty-based CL methods (UCB, UCL) and gives competitive numbers, beating its sampling-based predecessors and roughly matching HAT on most benchmarks (HAT still wins the eight-dataset sequence). If the KL issue is corrected and the experiments re-run with seeds, the empirical claim could stand.\n\nBottom line: this deserves a serious referee, because the method is concrete and the benchmark work is real. But I would not cite it in its current form; the core theory needs fixing first.","headline":"Moment-propagation CL is empirically serious, but the variational objective in Eq. 6 is mis-stated and the uncertainty story needs re-derivation.","tokens_in":13033,"tokens_out":3201,"would_cite":false,"duration_ms":29349,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learned parameter uncertainty alone can prevent catastrophic forgetting on task-incremental benchmarks.","keywords":["Continual learning","Catastrophic forgetting","Parameter uncertainty","Moment propagation","Variational inference","Learning rate adaptation","Bayesian regularization","Task incremental learning"],"falsifier":"Train an MP network on task A, rank parameters by inverse variance or SNR, then reinitialize or add noise to only the top-ranked \"important\" parameters while leaving the rest untouched. If accuracy on task A drops no more than when the same number of least-important parameters is disturbed, then the uncertainty signal is not identifying the parameters that carry task A's function, and the regularization mechanism in LRA and PPBI loses its justification.","tokens_in":1528,"feed_emoji":"🧠","tokens_out":2112,"duration_ms":74070,"temperature":0.7,"pith_summary":"The paper tries to establish that parameter uncertainty learned by a Moment Propagation network can act as a reliable importance signal for continual learning, letting a single network adapt to a stream of tasks without catastrophically forgetting earlier ones. It proposes two training regularizers—one that lowers learning rates for important parameters, one that raises the Bayesian KL penalty for them—and reports that both beat sampling-based Bayesian continual learning baselines and match the state of the art on several task-incremental benchmarks. A sympathetic reader should care because the approach replaces costly Monte Carlo sampling with a closed-form uncertainty estimate, potentially making uncertainty-driven continual learning practical for larger networks.","feed_headline":"Uncertainty signals reveal which weights to protect","feed_subtitle":"A sampling-free Bayesian method uses parameter variance to fight forgetting on seven task-incremental benchmarks.","key_machinery":"The Moment Propagation (MP) framework carries the argument. It is a variational-inference scheme where each network weight is a Normal random variable; the network propagates means and covariances through convolutional, linear, batch-normalization, and activation layers, using a first-order Taylor expansion of nonlinearities to get an analytical predictive mean and covariance. That yields a closed-form ELBO, so the network log-likelihood is directly differentiable and no Monte Carlo samples are needed. From the learned posterior, parameter importance is defined as either inverse variance $1/\\sigma_{\\omega_i}^2$ or signal-to-noise ratio $|\\mu_{\\omega_i}|/\\sigma_{\\omega_i}^2$, and these values are remapped across the whole network to per-parameter learning rates or per-parameter KL weights.","core_discovery":"The paper's central claim is that in a network trained with Moment Propagation, the posterior variance of each weight is a self-learned measure of that weight's functional importance: weights that matter for a task end up with small variance, while dispensable weights revert toward the standard-normal prior with variance near one. Using this map, the paper defines two continual-learning rules: Learning Rate Adaptation maps importance to per-parameter learning rates so critical weights move little, and Per-Parameter Bayesian Inference maps importance to per-parameter KL weights so critical weights are pulled back toward the previous task's posterior. The authors show that on seven task-incremental benchmarks, these two rules outperform the sampling-based UCB and UCL predecessors and land within a few points of Hard Attention to Task, without needing rehearsal, network expansion, or task-specific feature masks.","pith_inferences":["If the importance signal transfers across task orderings, the same uncertainty map could be reused as a task arrives later; reversing the task order and checking whether the protected parameters stay the same would test this directly.","Because MP gives a deterministic, differentiable covariance, the importance mask could be combined with feature-level freezing or masking approaches, potentially recovering HAT-like backward transfer without task-specific propagation masks.","The first-order Taylor approximation treats layer inputs as independent, so the importance measure ignores correlations between parameters; a second-order correction to the propagated covariance would reveal whether interaction effects materially change which parameters are flagged as important."],"forward_implications":["The two MP-based regularizers—Learning Rate Adaptation and Per-Parameter Bayesian Inference—improve average test accuracy and backward transfer over the sampling-based UCB and UCL methods on the evaluated benchmarks.","MP networks can identify unimportant parameters: around 95% of parameters are pushed toward the prior and can be pruned before accuracy drops, indicating that the uncertainty map is a usable importance signal.","On task-incremental split and permuted MNIST, Per-Parameter Bayesian Inference reaches 98–99% average test accuracy, while on split CIFAR10 and mixed CIFAR10/100, Learning Rate Adaptation reaches 77–92%.","The methods work without rehearsal, architecture expansion, or task-specific masks; they only adjust learning rates or KL weights according to the current importance map.","Feature freezing is a strong baseline: LRA and PPBI improve only marginally over hyperparameter-tuned freezing, except on mixed CIFAR10/100 where freezing underperforms."],"supporting_citations":[{"why":"Supplies the Moment Propagation method: propagating mean and covariance through layers with a first-order Taylor approximation, yielding the closed-form ELBO that produces the parameter uncertainties.","marker":"[18]"},{"why":"Introduces Learning Rate Adaptation with a sampling-based Bayesian network; the paper replaces the sampler with Moment Propagation and compares against it.","marker":"[24]"},{"why":"Provides the per-parameter KL regularization baseline (UCL) that Per-Parameter Bayesian Inference extends and compares against.","marker":"[25]"},{"why":"Defines Bayes-by-Backprop, the sampling-based variational inference approach used as the comparison framework in the pruning experiments.","marker":"[20]"},{"why":"Establishes Elastic Weight Consolidation, a standard regularization baseline for continual learning that the proposed methods must match or beat.","marker":"[2]"},{"why":"Introduces Hard Attention to Task, the only approach with stronger backward transfer on the sequence-of-eight benchmark, used as a comparison point.","marker":"[8]"}],"fun_headline_variants":["Weight variance maps critical parameters for continual learning","Sampling-free Bayesian approach uses parameter uncertainty to avoid forgetting","Variance-guided learning rates and KL weights protect old tasks","Moment propagation self-learns weight importance to defeat forgetting"],"cache_read_input_tokens":14976,"weakest_assumption_plain":"The load-bearing premise is that the variance produced by Moment Propagation's first-order Taylor approximation is an accurate and stable measure of a parameter's true contribution to the network's learned function, and that this importance ranking transfers from one task to the next; the pruning experiments test this only on a single task, not across a sequence.","fun_headline_variants_meta":{"raw":{"variants":["Weight variance maps critical parameters for continual learning","Sampling-free Bayesian approach uses parameter uncertainty to avoid forgetting","Variance-guided learning rates and KL weights protect old tasks","Moment propagation self-learns weight importance to defeat forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1473,"prompt_tokens":910,"completion_tokens":563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":499}},"tokens_in":526,"tokens_out":563,"duration_ms":6462,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:54:44.759130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an MP network on task A, rank parameters by inverse variance or SNR, then reinitialize or add noise to only the top-ranked \"important\" parameters while leaving the rest untouched. If accuracy on task A drops no more than when the same number of least-important parameters is disturbed, then the uncertainty signal is not identifying the parameters that carry task A's function, and the regularization mechanism in LRA and PPBI loses its justification.","supporting_citations":[{"cited_title":"PremiUm-CNN: Propagating uncertainty towards ro- bust convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the Moment Propagation method: propagating mean and covariance through layers with a first-order Taylor approximation, yielding the closed-form ELBO that produces the parameter uncertainties."},{"cited_title":"Uncertainty- guided continual learning with bayesian neural networks","cited_arxiv_id":null,"evidence_quote":"Introduces Learning Rate Adaptation with a sampling-based Bayesian network; the paper replaces the sampler with Moment Propagation and compares against it."},{"cited_title":"Uncertainty-based continual learning with adaptive regularization,","cited_arxiv_id":null,"evidence_quote":"Provides the per-parameter KL regularization baseline (UCL) that Per-Parameter Bayesian Inference extends and compares against."},{"cited_title":"Weight uncertainty in neural network,","cited_arxiv_id":null,"evidence_quote":"Defines Bayes-by-Backprop, the sampling-based variational inference approach used as the comparison framework in the pruning experiments."},{"cited_title":"Overcoming catastrophic forgetting in neural networks,","cited_arxiv_id":null,"evidence_quote":"Establishes Elastic Weight Consolidation, a standard regularization baseline for continual learning that the proposed methods must match or beat."},{"cited_title":"Overcoming catas- trophic forgetting with hard attention to the task","cited_arxiv_id":null,"evidence_quote":"Introduces Hard Attention to Task, the only approach with stronger backward transfer on the sequence-of-eight benchmark, used as a comparison point."}],"review_version":1}