{"id":"b8006696-7744-45d7-9874-77f6c81eec71","arxiv_id":"1908.01113","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Neural networks can be trained by the EnRML algorithm using covariance matrices instead of gradients, giving uncertainty estimates and robust small-data performance.","lead":"This paper describes Ensemble Neural Networks, a way to train neural networks without gradients by using covariance information from an ensemble of weight guesses. It claims strong performance on small data sets and built-in uncertainty estimates, beating Bayesian neural networks in several tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The covariance-to-gradient approximation in Eq. (10) is the load-bearing assumption; for ReLU networks with realistic ensemble spread it can be poor, and the paper provides no diagnostic or convergence support.","rationale":"The reader's weakest-assumption analysis correctly identifies the covariance-to-gradient approximation in Eq. (10) as the premise on which the paper's central claim depends. I agree with that identification. My reading adds specificity: the initial ensemble is deliberately broad, so the Taylor residual is not a small perturbation, and for ReLU networks the map is only piecewise linear, making the dropped terms in Appendix D potentially dominant. The paper presents no convergence analysis and no diagnostic for the approximation error. This is a genuine correctness risk, but it is not an internal contradiction in the sense of a false algebraic step: for linear maps and infinite ensembles the approximation is exact, and the synthetic experiments suggest it can work in practice. Therefore the right disposition remains CONDITIONAL, as the reader concluded: the method is plausible and the concern is addressable, but the missing validation and missing code mean the main claims should not yet be accepted as established. I would not move the verdict to REJECT because the derivation is a standard EnKF/EnRML approximation and the reported results, if reproducible, are nontrivial evidence.","tokens_in":36406,"tokens_out":8600,"duration_ms":96880,"concrete_test":"On the §3.2 ideal dataset and reference architecture, run the first five ENN iterations with N_e=100 and the default hyperparameters. At each iteration, after forming the ensemble covariances in Eq. (10), also compute the exact sensitivity matrix G for each realization by backpropagation (or finite differences) and form the exact Gauss-Newton update of Eq. (8) with the same λ. Measure (i) the relative Frobenius errors ||C_{M,D} - C_M G^T||_F / ||C_{M,D}||_F and ||C_D - G C_M G^T||_F / ||C_D||_F, and (ii) the cosine of the angle between the ENN update and the exact update, averaged over realizations. Repeat for a linear one-hidden-layer control, the reference 4-layer ReLU net, a 5-layer ReLU net, and initial weight variances N(0,1) versus N(0,0.1).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—a derivative-free update that simultaneously beats BNN—rests on Eq. (10). Appendix D justifies replacing the sensitivity matrix G by ensemble covariances through a first-order Taylor expansion, dropping residual terms e_j - e_bar in Eqs. (D.6) and (D.7). This requires the residual to be small over the ensemble. It is not: initial weights are sampled broadly (N(0,1); the supplementary m1 produces outputs differing several-fold across realizations), and a ReLU network is piecewise linear, so different realizations generically lie in different linear regions. No single G evaluated at the ensemble mean makes Eq. (D.3) accurate, and the neglected cross-terms can be as large as the retained covariance terms. The update direction in Eq. (10) can then be uncorrelated with, or opposite to, the true Gauss-Newton direction of Eq. (8), so convergence is not guaranteed. The advertised extension to non-differentiable neuron models and loss functions is especially affected, because the only derivation linking covariances to gradients assumes a differentiable map. The paper offers no convergence proof, no condition ensuring that Eq. (10) decreases the objective, and no empirical measurement of the approximation error on the tested architectures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ensemble Neural Networks (ENN), a training method for neural networks that replaces gradient-based backpropagation with an ensemble covariance update derived from the ensemble randomized maximum likelihood (EnRML) algorithm used in petroleum history matching. An ensemble of weight realizations is maintained; at each iteration the method computes cross-covariances between weights and predictions and the covariance of predictions, then updates all realizations through Eq. (10). The method is presented as gradient-free, Bayesian, naturally providing uncertainty quantification, and robust to small training data because the ensemble is said to enlarge the training set. Experiments include a one-dimensional toy regression, a synthetic 'ideal' regression problem, and four real-world datasets (three UCI plus one reservoir-simulation dataset), with comparisons against fully connected networks and three Bayesian neural network (BNN) variants.","tokens_in":36628,"tokens_out":7062,"duration_ms":72729,"significance":"If the central claims hold, the paper would offer a novel training paradigm for neural networks in small-data engineering settings, with built-in uncertainty estimates and no gradient computation. The connection between ensemble Kalman-filter-type methods and neural network training is timely and potentially valuable. The derivation is grounded in a mature inverse-modeling literature, and the supplementary material contains a hand-worked end-to-end example, which is a genuine reproducibility aid. The synthetic experiments in Section 3.2 use 50 independent runs and boxplots, which is good practice. However, the load-bearing covariance-to-gradient substitution is only heuristically justified, and the real-world experiments lack statistical rigor and adequate baseline specification.","major_comments":[{"comment":"The covariance-to-gradient substitution in Eq. (9) is the load-bearing step that converts the Gauss-Newton update (8) into the derivative-free update (10). Appendix D derives it by dropping all terms involving e_j - e_bar in Eqs. (D.6) and (D.7), justified by asserting that the first-order Taylor residual is small. For a ReLU network with weights initialized from a standard normal distribution, as in the Supplementary Material, the ensemble members at early iterations generically lie in different linear regions of the piecewise-linear map g(m), so a single sensitivity matrix G evaluated at the ensemble mean does not make the residual small; the neglected cross-terms can be of the same order as the retained covariance terms. Consequently the update direction in Eq. (10) need not approximate the direction of Eq. (8), and there is no guarantee that the objective (3) decreases. The paper contains no diagnostic measuring the approximation error on the architectures tested and no convergence proof. This is a central gap for the 'gradient-free' claim.","section":"Section 2.2, Eq. (9) and Appendix D"},{"comment":"The paper motivates gradient-free training by citing Hodgkin-Huxley neuron models and non-differentiable losses such as BLEU, and the Discussion repeats that the ENN is suitable for these settings. However, the only derivation connecting covariances to the update direction is the first-order Taylor expansion in Appendix D, which presupposes that g(m) is differentiable with respect to m. For non-differentiable neuron models or losses, the substitution in Eq. (9) has no theoretical basis, and no experiment with such models or losses is reported. The claim that the ENN 'enables the use of complicated neuron models and loss functions' is therefore not supported.","section":"Section 1 and Section 4"},{"comment":"Table 3 reports average estimation losses for the real-world datasets without error bars, number of trials, or the train/test splitting protocol; it is not stated whether each entry is a single run or an average over many. The BNN baselines are not adequately specified: the prior variance, learning rate, number of iterations, and the validation procedure for BNN-Val are not given, so the reader cannot assess whether the comparison is favorable to the ENN by construction. No code is provided, which further limits reproducibility. As a result, the abstract's claim that 'the ENN performs much better than the traditional Bayesian neural networks' is not statistically established by the real-world experiments.","section":"Section 3.3, Table 3"},{"comment":"The paper repeatedly attributes the small-data robustness of the ENN to the ensemble 'enlarging the training dataset' through duplication with random disturbance. This is only a heuristic: all realizations use the same input vectors and differ only in perturbed targets and initial weights, so no new information about the input-output mapping is introduced. The claimed mechanism is not quantified (e.g., no effective sample size or bias-variance analysis), and an alternative explanation is the regularizing prior term in Eq. (3). The authors should either provide a formal argument for this mechanism or temper the claim.","section":"Section 2.3 and Section 4"},{"comment":"The toy experiment demonstrates uncertainty bands qualitatively, but the paper does not quantitatively assess the quality of the uncertainty estimates (e.g., empirical coverage of the stated credible intervals, or sharpness of the predictive distributions). Given that uncertainty quantification is one of the primary advertised advantages of the ENN, the absence of any calibration metric leaves this claim unverified.","section":"Section 3.1"}],"minor_comments":[{"comment":"The text says 'the two equivalent equations in Eq. (6) and Eq. (7)' but the identities being derived are Eq. (5) and Eq. (6) of Section 2.2; the cross-references should be corrected.","section":"Appendix C"},{"comment":"The loss metric used in Table 3 and Figures 7-9 is not defined in the main text; the supplementary material defines an absolute-error loss in Eq. (S.1), but it is unclear whether the real-world results use the same normalization.","section":"Section 3 and Supplementary Material"},{"comment":"The caption states that the gray area corresponds to three standard deviations, but the text does not specify whether this is the predictive uncertainty or the parameter uncertainty; this distinction is important for the uncertainty-quantification claims.","section":"Figure 3"},{"comment":"The values of the lower bound of lambda (0.005) and the factor gamma (10) are stated as fixed, but no sensitivity analysis is provided, so the reader cannot judge how the results depend on these choices.","section":"Appendix B"},{"comment":"The claim that the ENN is 'simpler' than the BNN is not quantified; no runtime, memory, or iteration-count comparisons are reported.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript combines a well-established ensemble assimilation method with neural network training; the idea is interesting, but the evaluation falls short of the standards expected in a machine learning venue. The Appendix D covariance approximation is the main technical risk, and the real-world comparison is under-specified. I would support a major revision that includes a diagnostic of the covariance-to-gradient approximation, improved BNN baselines with error bars, a quantitative uncertainty-calibration check, and code release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is straightforward: the authors take the ensemble randomized maximum likelihood algorithm from history matching and use it to train neural networks without gradients. That port is real and the paper is the first to make it, as far as I can tell. The Bayesian framing is standard, and the synthetic experiments do show the method can converge on small-data problems, with the BNN comparison becoming fair once enough data are available. The supplementary worked example is a nice touch, and the paper is honest that the BNN catches up when data are plentiful.\n\nThe soft spot is the one the stress-test note points to: Eq. (10) replaces the sensitivity matrix G with ensemble covariances using a first-order Taylor expansion, and Appendix D simply asserts that the residual terms are small. For deep ReLU networks with broad initial weight distributions, that is not obviously true, and the paper gives no diagnostic, no convergence proof, and no measurement of the approximation error on the architectures it tests. That is a real limitation, though not necessarily a fatal one. The method works in the experiments, which suggests the approximation is adequate in those settings, but the advertised extension to non-differentiable neuron models is especially suspect because the only derivation assumes a differentiable map.\n\nThe rest of the evidence is also thinner than the prose. Table 3 reports single runs on real data with no error bars, the BNN hyperparameters are not specified, and the \"ensemble enlarges the training data\" claim is heuristic rather than demonstrated. These are addressable weaknesses, not conceptual ones.\n\nWho gets value from this paper? Anyone working on derivative-free training, small-data uncertainty quantification, or bringing ensemble smoothers into machine learning. It deserves a serious referee, because the idea is novel and the central claim is plausible, but the referee should insist on an empirical study of the covariance approximation and a fuller comparison with proper error bars.","headline":"A legitimate new application of EnRML to neural network training, but the covariance-to-gradient approximation is asserted rather than validated and the real-data comparisons are thin.","tokens_in":37148,"tokens_out":1755,"would_cite":false,"duration_ms":22113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","65C05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that neural networks can be trained with no gradient computations at all, by replacing the sensitivity matrix with covariances estimated from an ensemble of weight realizations, and that this yields uncertainty estimates…","keywords":["ensemble neural networks","gradient-free optimization","ensemble randomized maximum likelihood","uncertainty quantification","Bayesian neural networks","small training data","covariance matrices","stochastic realizations"],"falsifier":"Take a regression problem with a deliberately nonlinear activation and initialize the ensemble with wide weight distributions; compute the sample cross-covariance $C_{M,D}$ and compare it against the product $C_M G^T$ using finite-difference gradients. If the relative discrepancy is large in early iterations and the ENN update fails to reduce test loss while gradient descent succeeds, the linearity assumption behind Eq. (10) is the point of failure.","tokens_in":36207,"feed_emoji":"🧠","tokens_out":9182,"duration_ms":89850,"temperature":0.7,"pith_summary":"Ensemble neural networks (ENN) is a proposed training method that removes the gradient from neural network optimization entirely: an ensemble of weight realizations is updated using covariance statistics, and those same realizations supply predictive uncertainty. The method is built on the Bayesian posterior over weights and uses the ensemble randomized maximum likelihood (EnRML) algorithm from inverse modeling as the optimizer. The paper's central experimental claim is that ENN trains more accurately than traditional Bayesian neural networks on small datasets, while also handling non-differentiable loss functions and neuron models. This matters because real engineering problems often have expensive, scarce data and physical models whose derivatives are hard or impossible to obtain. If correct, ENN would give practitioners a way to train neural networks with uncertainty quantification and without backpropagation.","feed_headline":"Neural networks can train without gradients using ensemble covariances","feed_subtitle":"A covariance-based optimizer replaces backpropagation and outperforms Bayesian neural networks on small datasets.","key_machinery":"The load-bearing mechanism is the EnRML update of Eq. (10). EnRML is an ensemble-based inverse modeling algorithm that iteratively adjusts a set of parameter realizations to match observed data; here the parameters are the neural network weights and the forward map is the network's feed-forward computation. Instead of forming the sensitivity matrix $G$, it computes the cross-covariance $C_{M,D}^l$ between the current weight realizations and their predicted outputs, and the covariance $C_D^l$ of predicted outputs, and uses these in place of $G$ inside a Gauss-Newton step. A multiplier $\\lambda$ regulates the step size: it grows when the ensemble's data mismatch worsens and shrinks when the mismatch improves, preventing over-updates early in training. The same ensemble of realizations thus does double duty: it provides the covariance statistics that drive optimization and, at convergence, provides the spread that gives predictive uncertainty intervals.","core_discovery":"The discovery the paper argues for is that the Gauss-Newton update for a Bayesian neural-network objective can be made gradient-free by substituting ensemble covariances for the sensitivity matrix. Starting from the posterior $p(m|d_{\\mathrm{obs}})\\propto \\exp[-O(m)]$, the objective $O(m)=\\frac{1}{2}(m-m_{pr})^T C_M^{-1}(m-m_{pr})+\\frac{1}{2}(g(m)-d_{\\mathrm{obs}})^T C_D^{-1}(g(m)-d_{\\mathrm{obs}})$ is minimized by repeated application of Eq. (10), in which $C_{M,D}^l$ (cross-covariance between weights and predicted outputs) and $C_D^l$ (covariance of predicted outputs) replace $G$. The justification is the first-order Taylor approximation $C_{M,D}=C_M G^T$, $C_D=G C_M G^T$, with residual terms neglected. The resulting update moves every realization toward its own perturbed observation, and the spread of the converged ensemble acts as the posterior uncertainty. The paper reports that this scheme converges on a synthetic problem with 70 training points and 93 weights, and that it achieves lower loss than three Bayesian neural network baselines across four real-world datasets.","pith_inferences":["A testable extension the paper does not run is to vary the ensemble size systematically; one would expect a bias-variance trade-off in the covariance proxy, with too few realizations making the update noisy and too many adding little.","The covariance proxy suggests a direct link to natural-gradient and ensemble Kalman methods; if the approximation holds, any ensemble smoother could serve as a generic neural optimizer, not only EnRML.","The paper compares only with traditional BNN; a fairer modern benchmark would include scalable approximate inference, and the distinctive claim of being gradient-free would survive even if the accuracy margin does not."],"forward_implications":["Because no derivative of the network output is ever needed, the ENN update is compatible with neuron models and loss functions that are not differentiable, including biophysical neuron models and discrete evaluation metrics.","The ensemble construction acts as implicit data augmentation, so ENN can train networks with more weights than training examples; the paper shows convergence with 70 training points and 93 weights and lower loss than BNN when data-to-weight ratios are small.","Uncertainty quantification is a by-product of the optimization: the distribution of converged weight realizations directly supplies posterior predictive intervals, without a separate inference pass.","With abundant training data the benefit shrinks: the paper observes that the BNN's loss approaches the ENN's as data grow, and a plain fully connected network can beat both when data are plentiful, so the practical advantage is concentrated in the small-data, uncertainty-requiring, gradient-free regime.","Since the forward pass is unchanged, the same EnRML step can replace gradient descent in other architectures, including CNNs and RNNs, whenever their forward computations are well defined."],"supporting_citations":[{"why":"Introduces the EnRML algorithm, the covariance-based ensemble optimizer that ENN adopts as its training loop.","marker":"Gu and Oliver (2007)"},{"why":"Supplies the Levenberg-Marquardt step-size control for the iterative ensemble update, including the multiplier lambda.","marker":"Chen and Oliver (2013)"},{"why":"Provides the Bayesian inverse-theory formulation and the ensemble update equations that Eq. (10) is built on.","marker":"Oliver et al. (2008)"},{"why":"Gives the covariance-approximation identities used to replace gradients with the cross-covariance and data covariance.","marker":"Reynolds et al. (2006)"},{"why":"Contributes the toy regression benchmark used to evaluate uncertainty quantification performance.","marker":"Hernandez-Lobato and Adams (2015)"},{"why":"Defines the BNN machinery (objective, hyperparameters, Hessian approximations) that ENN is compared against.","marker":"Bishop (2007)"}],"fun_headline_variants":["Train neural networks without gradients via ensemble covariances","Gradient-free neural network training with covariance matrices","Ensemble neural networks: no backprop, no gradients","Covariance-based optimizer replaces backpropagation in neural nets","ENN: gradient-free Bayesian training beats traditional BNN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the network's outputs are nearly linear in the weights across the ensemble's spread, so the sample covariance between weights and outputs equals the weight covariance times the gradient; if that linearity fails, the covariance proxy points the update in the wrong direction and convergence is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Train neural networks without gradients via ensemble covariances","Gradient-free neural network training with covariance matrices","Ensemble neural networks: no backprop, no gradients","Covariance-based optimizer replaces backpropagation in neural nets","ENN: gradient-free Bayesian training beats traditional BNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3865,"prompt_tokens":1023,"completion_tokens":2842,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2763}},"tokens_in":639,"tokens_out":2842,"duration_ms":20432,"temperature":1.0,"reasoning_tokens":2763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:22:41.828480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a regression problem with a deliberately nonlinear activation and initialize the ensemble with wide weight distributions; compute the sample cross-covariance $C_{M,D}$ and compare it against the product $C_M G^T$ using finite-difference gradients. If the relative discrepancy is large in early iterations and the ENN update fails to reduce test loss while gradient descent succeeds, the linearity assumption behind Eq. (10) is the point of failure.","supporting_citations":[],"review_version":1}