{"id":"de6b7608-6f17-412c-a53f-212578434833","arxiv_id":"2411.14202","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"The ECL-RR method combines a revised mean/variance regularizer, a parameter-learning-network compression, and a common/distinctive subspace decomposition, and reports top accuracy on four continual learning benchmarks.","lead":"This paper presents a continual learning method for Bayesian neural networks that replaces the standard KL regularizer with custom mean and variance penalties, compresses stored weights with a learned generator network, and partitions parameters into common and task-specific subspaces. The authors report higher average accuracy than several published baselines on Permuted MNIST, Split MNIST, alternating CIFAR 10/100, and an 8-mixture dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dense-layer storage claim rests on a 'shared trajectory' assumption that is false in a standard MLP: output nodes receive identical inputs but different backpropagated errors, so their weight rows do not move together; if this premise fails, the PLN cannot generate the layer and the claimed…","rationale":"The reader's weakest_assumption is exactly the premise I find most load-bearing. The paper's strongest claim bundles a performance result with a storage-reduction mechanism; the storage claim cannot survive without Section 4.1's shared-trajectory premise. Because in a fully connected layer the per-node gradient uses dL/dy_j, which is not shared, the premise is likely false, and the paper provides no quantitative support. I also note the Eq. 4/5 derivations flagged by the reader are genuinely incorrect and further raise correctness risk, but the trajectory assumption is the cleaner single point: a direct measurement can falsify it, and unlike a typo in a derivation it attacks the algorithmic mechanism rather than only its justification. The absence of code, seeds, and error bars compounds the issue, but the trajectory check is sufficient to maintain the reject verdict.","tokens_in":14475,"tokens_out":10146,"duration_ms":106822,"concrete_test":"Train the proposed DCN on Permuted MNIST following Section 5. After each session, store the m rows of a chosen dense layer and compute the mean row trajectory across nodes and the spread around it (e.g., average normalized distance from each row to the row-mean). If the spread is comparable to or larger than the average between-session drift, the Section 4.1 'consistent trajectory' premise is empirically false. This single measurement decides whether a single PLN can reasonably generate all m rows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 asserts that 'every k-dimensional tensor with size m shares a consistent trajectory in parameter space' because all outgoing nodes observe the same input nodes, so 'any perturbation on the incoming side equally affects all outgoing nodes' and 'all connections need to be updated similarly.' This is not true for backpropagation. In a fully connected layer y_j = sum_i W_{ji} x_i, the update to row j is proportional to (dL/dy_j) x_i; the factor dL/dy_j is different for every output node j, so even with identical inputs the rows receive different gradient signals. The PLN as described (G_psi: R^k -> R^k, one random sample to parameter space) has no mechanism to generate m distinct row trajectories. If those trajectories diverge, one shared distribution generator cannot reconstruct the layer, and the claimed 'size of PLN << R^k' storage reduction has no basis. The section also provides no parameter count for the PLN and no quantitative measurement of trajectory overlap; the GTM visualization is qualitative. Section 4.2's analogous 'overlapping pattern' for conv masks is asserted without statistical support. This is load-bearing because efficient storage is one of the two headline contributions, and the experimental tables do not report whether the PLN was actually used to generate the evaluated networks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Bayesian variational-inference method for continual learning, called ECL-RR. Its components are: (i) a revised regularizer in Eq. (4) that replaces the KL term with positive penalties on the mean difference and on variance magnitudes; (ii) a Parameter Learning Network (PLN) that is claimed to store the dense-layer posterior compactly by exploiting a shared parameter trajectory across output nodes (Sec. 4.1); (iii) an SVD-based decomposition of representation matrices into common and distinctive subspaces with a claimed backward-knowledge-transfer condition (Eq. 7); and (iv) an importance-weighted LCOR term (Eq. 5) meant to capture parameter-data correspondence. Experiments report state-of-the-art average accuracy and backward transfer on Permuted MNIST, Split MNIST, alternating CIFAR 10/100, and an 8-mixture dataset (Table 1).","tokens_in":14778,"tokens_out":7527,"duration_ms":71802,"significance":"If the theoretical and architectural claims were correct, the paper would address two genuinely important problems in continual learning: compact storage of Bayesian posteriors and backward knowledge transfer. The broad benchmark comparison is a useful feature, and the GTM visualizations are a reasonable way to inspect parameter drift. However, the paper does not provide machine-checked proofs, reproducible code, or storage measurements, and several load-bearing derivations and architectural premises are incorrect. As it stands, the efficient-storage contribution and the regularization contribution are not established on the evidence in the manuscript.","major_comments":[{"comment":"The displayed gradient is not the gradient of KL(Q(W_t|theta_t) || Q(W_{t-1}|theta_{t-1})). For diagonal Gaussian variational distributions, the derivative with respect to mu_t is Sigma_{t-1}^{-1}(mu_t - mu_{t-1}), not sigma_t^{-1}(mu_t - mu_{t-1}); the variance derivative also does not contain the mean-difference term shown, because the mean term in the KL uses the previous covariance. The two regularization terms in Eq. (4) are therefore not derived from the stated KL gradient, and the paper's explanation that sigma_t controls the sensitivity of the mean update is based on a wrong derivative.","section":"Gradient Analysis preceding Eq. (4)"},{"comment":"The claim that positive regularizers ensure the loss remains a lower bound of the log evidence is incorrect as stated. The objective in Eq. (3) is the negative ELBO, which is an upper bound on -log P(D), not a lower bound on log P(D). Adding arbitrary positive terms to Eq. (3) breaks any ELBO lower-bound interpretation unless a separate derivation is supplied. This invalidates the theoretical justification given for the revised objective.","section":"Paragraph after Eq. (4)"},{"comment":"The premise that 'every k-dimensional tensor with size m shares a consistent trajectory in parameter space' is false for a standard fully connected layer. In a layer with output y_j = sum_i W_{ji} x_i, the gradient for row j is proportional to (dL/dy_j) x^T; the input x is shared, but dL/dy_j differs across output nodes, so rows receive different updates. A PLN G_psi: R^k -> R^k mapping one random sample to one parameter vector has no mechanism to generate m distinct row trajectories. Since the dense-layer storage claim depends entirely on this premise, the statement 'size of PLN << R^k' is unsupported, and the manuscript provides no parameter counts or quantitative trajectory-overlap statistics.","section":"Section 4.1"},{"comment":"The backward-transfer evaluation is circular. The common and distinctive subspace bases are obtained by SVD on representation matrices built from the same session parameter histories that are later used to evaluate Eq. (7) and to compute BWT, and the correlation gate in Eq. (6) uses the LCOR objective that the method is optimizing. A positive BWT (Table 1d: 0.24) can therefore reflect self-selection rather than genuine cross-session transfer. The paper needs a held-out basis-selection procedure or an ablation with randomly chosen bases to support the inequality in Eq. (7).","section":"Section 4.2 and Eq. (7)"},{"comment":"The headline efficient-storage contribution is not empirically validated. The tables report only ACC and BWT; they do not report the parameter count of the PLN, whether the evaluated networks were actually generated by the PLN, or whether the reported accuracy is obtained from the PLN samples. Without this information, the claim that ECL-RR 'significantly reduces storage requirements' is not supported by the experiments. The tables also omit standard deviations and the number of runs, so the margins over HAT and UCB (e.g., 0.38% on Permuted MNIST, 0.6% on the 8-mixture dataset) cannot be assessed.","section":"Section 5 and Table 1"},{"comment":"The importance-weighted estimator R_{t,M} = (1/M) sum_i Q(W_{t-1}^i, D_{t-1}) / Q(W_t^i) is not a standard importance-weighted ELBO for the current session, and the paper does not define the joint proposal Q(W,D) or show that E[log R_{t,M}] is a lower bound for log P(D_t). The additional KL terms in Eq. (5) also appear without a derivation. As written, the LCOR objective is not a well-defined estimator of log evidence.","section":"Eq. (5)"}],"minor_comments":[{"comment":"The equation reference is left as 'as shown in ??', which should be fixed.","section":"Eq. (2)"},{"comment":"The reported HAT accuracy is inconsistent: the text first states HAT achieves 91.6%, while Table 1a lists 97.3% and the later sentence cites 97.34%.","section":"Table 1a and Section 5.2"},{"comment":"The correlation coefficient rho and the threshold epsilon are not defined, and no values are given for epsilon in the experimental section.","section":"Eq. (6)"},{"comment":"Both subcaptions (a) and (b) describe 'convolutional neural network' even though panel (a) is described in the text as the fully connected case; this is confusing.","section":"Figure 1"},{"comment":"The '8-mixture dataset' is not described; the reader cannot tell how many sessions it contains or how the mixture is generated.","section":"Section 5.1"},{"comment":"The notation for elementwise square and division in terms such as (mu_t - mu_{t-1})^2 / sigma_t^2 is ambiguous for tensor parameters and should be defined explicitly.","section":"Eq. (4)"},{"comment":"The statement that layer-wise correlation is 'independent of the dataset' is asserted without any proof or measurement; this is a strong claim that needs support.","section":"Section 4"}],"recommendation":"reject","confidential_remarks":"The manuscript has a substantial empirical section, but the two central theoretical contributions -- the revised KL-based regularizer and the shared-trajectory storage argument -- rest on incorrect derivations. Correcting them is not a local fix: the gradient in Section 4 would need to be recomputed, the regularizer re-derived or replaced, and the PLN architecture and its experiments rethought. In addition, the storage claim is never actually tested. I therefore do not see a path to acceptance within the current manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a paper with competitive continual-learning numbers and a few genuinely interesting pieces, but the theoretical core is not sound. The KL gradient that motivates the revised regularizer is wrong, the importance-weighted ELBO is not the IWAE construction it cites, and the storage-efficiency claim depends on an assumption about parameter trajectories that does not hold for a standard MLP.\n\nWhat is new: the specific combination of (a) mean/variance penalties that replace KL in the variational loss, (b) a small parameter-learning network (PLN) to generate dense-layer weights instead of storing them, and (c) an SVD-based partition of the parameter space into common and distinctive subspaces for backward transfer. On the face of it, the reported accuracies are strong: 97.8% Permuted MNIST, 99.7% Split MNIST, 80.16% alternating CIFAR 10/100, 84.58% on 8-mixture, all close to or above the strongest baselines they compare against.\n\nThe soft spots are serious, though. The gradient of KL between successive Gaussian posteriors is misstated: the gradient w.r.t. mu_t is (mu_t - mu_{t-1})/sigma_{t-1}^2, not sigma_t^{-1}(mu_t - mu_{t-1}). The variance gradient is also wrong. Since the proposed mean/variance regularizers are supposed to be \"revised KL,\" the derivation doesn't support the method. The claim that positive regularizers keep the loss a lower bound of the log evidence is inverted reasoning. The IW-ELBO expression in Eq. 5 is not the IWAE bound from Domke & Sheldon or Sobolev & Vetrov, and the paper does not define terms carefully enough to be checkable.\n\nThe efficiency story has a load-bearing flaw that the paper states as a fact: \"every k-dimensional tensor with size m shares a consistent trajectory in parameter space.\" For a fully connected layer, the row updates are (dL/dy_j) * x_i, and dL/dy_j differs per output node, so rows do not move together. A single PLN with one input-to-parameter map has no way to generate m distinct trajectories. The paper reports no parameter count for the PLN, no quantitative measure of trajectory overlap (the GTM plot is qualitative), and nothing confirming whether the PLN was actually used in the networks whose accuracy is reported.\n\nExperimental reporting is thin: no seeds, no error bars, no code, and the Permuted MNIST text quotes UCB at 91.44% and 97.42% in adjacent sentences; the table uses 97.42. These omissions are not fatal by themselves, but they matter here because the headline is \"we outperform all baselines by a margin.\"\n\nBottom line: the empirical idea—regularize mean/variance separately, compress dense layers with a generative network, decompose the parameter space into shared and task-specific bases—is worth a second look. As written, the math and the storage claim do not hold up. A serious referee would send it back for major revision: fix the KL derivation or drop the claim that the regularizer is derived from it, prove or at least measure the trajectory assumption, and release code plus multiple seeds. I would not cite it in its current form, but I would not desk-reject it either; the experiments are strong enough that a corrected version could matter.","headline":"Interesting empirical continual-learning recipe whose theoretical justification and storage claims both break under scrutiny; worth a major revision, not a desk reject.","tokens_in":15283,"tokens_out":3736,"would_cite":false,"duration_ms":31430,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that its revised Bayesian regularizer, which learns parameter distributions and splits the weight space into common and distinctive subspaces, outperforms prior continual-learning methods while storing far fewer parameters.","keywords":["continual learning","Bayesian neural networks","variational inference","catastrophic forgetting","knowledge transfer","parameter distribution learning","common and distinctive subspaces","importance-weighted ELBO"],"falsifier":"Measure, during actual continual training on one of the reported benchmarks, the pairwise distances or cosine similarities of the per-node $k$-dimensional weight vectors in a dense layer across sessions. If those vectors do not move as a tight cluster, with per-session displacement variance comparable to the mean displacement, then a single Parameter Learning Network cannot faithfully regenerate all tensors from one distribution, and the reported storage reduction would not transfer to settings where trajectories diverge.","tokens_in":14200,"feed_emoji":"🧠","tokens_out":6476,"duration_ms":56165,"temperature":0.7,"pith_summary":"The paper tries to establish that a Bayesian neural network can learn a sequence of tasks without catastrophic forgetting by replacing the standard KL-divergence regularizer with revised penalties that control how the mean and variance of each parameter move. It adds a small Parameter Learning Network that generates dense-layer weights from a shared distribution, plus a representation matrix for convolutional layers, so old-task parameters do not have to be stored in full. The parameter space is split into common and distinctive subspace bases via SVD, and an importance-weighted ELBO aligns parameters with data across sessions. If the claims hold, continual learners get higher average accuracy and backward knowledge transfer while using far less memory than standard Bayesian continual learning.","feed_headline":"New Bayesian regularizer tops four continual-learning benchmarks","feed_subtitle":"Replacing the KL penalty with mean-variance terms and shared subspace bases improves accuracy and cuts parameter storage","key_machinery":"The central objects are the Parameter Learning Network (PLN), a small network $G_\\psi: \\mathbb{R}^k \\to \\mathbb{R}^k$ that generates all dense-layer weight tensors from one learned distribution under the assumption that all $k$-dimensional tensors of size $m$ follow a shared parameter trajectory, and the revised regularizer that replaces the KL term with $\\|\\mu_t-\\mu_{t-1}\\|_2^2 + \\left\\|\\frac{(\\mu_t-\\mu_{t-1})^2}{\\sigma_t^2}\\right\\|_2^2 + \\|\\sigma_{t-1}\\|_1 + \\left\\|\\frac{\\sigma_t-\\sigma_{t-1}}{\\sigma_{t-1}}\\right\\|_1$. For convolutional layers, flattened masks are stored in representation matrices, and SVD partitions the parameter space into common and distinctive subspace bases. The importance-weighted ELBO $L_{COR}$ uses the ratio $R_{t,M} = \\frac{1}{M}\\sum_i \\frac{Q(W_{t-1}^i, D_{t-1})}{Q(W_t^i)}$ to correlate parameters with past data, and the correlation threshold decides when basis reuse is safe for backward knowledge transfer.","core_discovery":"The central claim is that the proposed ECL-RR outperforms state-of-the-art Bayesian and non-Bayesian continual learning methods on Permuted MNIST (97.8% accuracy), Split MNIST (99.7%), alternating CIFAR 10/100 (80.16%), and the 8-mixture dataset (84.58%), while storing only the Parameter Learning Network and subspace bases instead of full per-session parameter tensors. The mechanism is a revised loss in which the KL term is replaced by four penalties: two on mean movement (raw magnitude and magnitude relative to current variance) and two on variance (sparsity of past variance and boundedness of relative variance change). A correlation-based importance-weighted ELBO then identifies which subspace bases are common across sessions and which are distinctive, enabling backward transfer: reusing common bases for the current task and adding new distinctive bases when the correlation check fails.","pith_inferences":["The shared-trajectory assumption could be tested on transformer or large-scale vision continual learning, where nodes are less symmetric; if trajectories diverge per node, a single PLN per layer would need to be replaced by per-cluster generators, degrading storage savings gracefully.","The common and distinctive subspace idea can be recast as a low-rank plus sparse decomposition of the representation matrix, where backward transfer corresponds to keeping the common low-rank component fixed and appending sparse novel components.","The variance-sparsity regularizer implies an interpretable uncertainty budget: nodes that become certain early stop moving, while nodes that remain uncertain stay learnable; one could predict per-task forgetting by tracking which nodes stay in the high-variance regime."],"forward_implications":["If ECL-RR works as claimed, replacing KL with mean-variance penalties should reduce catastrophic forgetting under sparse past-data access, because the gradient issues tied to KL's scale factor are bypassed.","Parameter storage for Bayesian continual learning can fall from twice the network size plus per-session copies to the size of the PLN plus subspace bases, with little or no accuracy loss on the reported benchmarks.","The common and distinctive subspace partition provides backward knowledge transfer when the correlation between the current loss and the previous-session loss exceeds a threshold.","The importance-weighted ELBO gives a tighter lower bound than plain ELBO, offering a principled way to align parameters with earlier-session data without replaying all past samples."],"supporting_citations":[{"why":"Supplies Bayes by Backprop, the Gaussian mean-variance variational parameterization that the paper adapts.","marker":"[5]"},{"why":"VCL is the main Bayesian continual learning baseline and supplies the coreset idea used for backward transfer evaluation.","marker":"[27]"},{"why":"Uncertainty-guided continual learning with Bayesian neural networks is the closest baseline the paper claims to surpass on accuracy.","marker":"[11]"},{"why":"Hard Attention to the Task is the strongest non-Bayesian baseline on several benchmarks and the accuracy point ECL-RR must beat.","marker":"[35]"},{"why":"Importance weighting for variational inference underlies the correlation-based ELBO that aligns parameters with past data.","marker":"[10]"},{"why":"Importance-weighted hierarchical variational inference motivates the log-evidence approximation used in the correlation term.","marker":"[38]"},{"why":"VCL-GNG and VCL-Vadam, two variational continual learning variants, are baselines for the empirical comparisons.","marker":"[39]"}],"fun_headline_variants":["New regularizer boosts continual learning and slashes storage","Bayesian continual learning gets a storage-friendly boost","Less memory, better recall: new Bayesian method for continual learning","Correlation-aware Bayesian regularizer for efficient continual learning","Bayesian regularizer: less storage, more knowledge retention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole storage reduction rests on the assumption that all weight vectors in a layer drift in nearly the same way during training, so one small network can generate all of them; the paper offers visual evidence but no quantitative proof.","fun_headline_variants_meta":{"raw":{"variants":["New regularizer boosts continual learning and slashes storage","Bayesian continual learning gets a storage-friendly boost","Less memory, better recall: new Bayesian method for continual learning","Correlation-aware Bayesian regularizer for efficient continual learning","Bayesian regularizer: less storage, more knowledge retention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2836,"prompt_tokens":977,"completion_tokens":1859,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1781}},"tokens_in":593,"tokens_out":1859,"duration_ms":13382,"temperature":1.0,"reasoning_tokens":1781,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:25:56.509815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, during actual continual training on one of the reported benchmarks, the pairwise distances or cosine similarities of the per-node $k$-dimensional weight vectors in a dense layer across sessions. If those vectors do not move as a tight cluster, with per-session displacement variance comparable to the mean displacement, then a single Parameter Learning Network cannot faithfully regenerate all tensors from one distribution, and the reported storage reduction would not transfer to settings where trajectories diverge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Uncertainty-guided continual learning with Bayesian neural networks is the closest baseline the paper claims to surpass on accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hard Attention to the Task is the strongest non-Bayesian baseline on several benchmarks and the accuracy point ECL-RR must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Importance weighting for variational inference underlies the correlation-based ELBO that aligns parameters with past data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Importance-weighted hierarchical variational inference motivates the log-evidence approximation used in the correlation term."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VCL-GNG and VCL-Vadam, two variational continual learning variants, are baselines for the empirical comparisons."}],"review_version":1}