REVIEW 3 major objections 4 minor 12 references
A Constructive Approach for Data-Driven Randomized Learning of Feedforward Neural Networks
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Single-hidden-layer networks can be grown by keeping only sigmoids that reduce training error.
desk verdict A clean, incremental constructive extension of the author's own D-DM that shows real compactness gains on four regressions, but its headline SOTA claim rests on an unpublished self-cited predecessor and an untested residual assumption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is an adaptive acceptance threshold $\theta$ wrapped around a data-driven candidate generator. Each candidate sigmoid is anchored at a random training point $x^*$; a hyperplane fitted to the point and its $k$ nearest neighbours gives local slope coefficients $a'_j$, and the sigmoid weights are set to $a_j = 4a'_j$ with bias $b = -a^T x^*$, placing the sigmoid's inflection point at $x^*$. The candidate is added only if the training error decreases by at least $\theta$; after $Q$ consecutive rejections $\theta$ is halved, so early stages admit only coarse-shape nodes and later stages admit detail nodes. The output weights are re-estimated by a standard linear least-squares fit through the pseudoinverse of the hidden-layer output matrix after every accepted node.
What would settle it
Run the paper's 100-trial protocol on function (6) with the acceptance filter disabled, so every candidate is kept while all other settings are unchanged; if CD-DM does not need far fewer nodes than D-DM to reach RMSE ≈ 0.00068 on the held-out test points, the filter is not the source of the reported compactness.
Extended reading notes
Core claim
The central claim is that an acceptance filter on incremental error reduction turns a randomized single-hidden-layer network into a compact constructive model without sacrificing accuracy. In the proposed CD-DM, candidate sigmoids are generated exactly as in the original data-driven method, but instead of being added wholesale they are admitted only when the training root mean square error decreases by at least $\theta$; $\theta$ is halved once $Q$ consecutive candidates fail, so the network first locks in nodes that capture the target's rough shape and then fills in detail. The reported evidence is that on function (6) a 33-node CD-DM reaches about the same test error ($0.00068$) as a 250-node D-DM, and on Stock, Concrete and Compactiv the constructive version needs two to five times fewer nodes and has lower median test error with less dispersion across runs. The paper presents this as a general statement: architecture size should be governed by target-function complexity, not by a fixed draw of random parameters.
Load-bearing premise
The method assumes that repeatedly sampling hyperplanes fitted to the original target function's local neighborhoods, and keeping only candidates that lower training error, is enough to approximate the residual detail that earlier nodes missed; the paper does not test a residual-based generator that would make this assumption unnecessary.
Editorial extensions
If this is right
- The random-parameter interval ceases to be a tuning problem: weights and biases come from local data geometry, and model selection reduces to choosing the final node count and the neighborhood size.
- Because only accepted nodes enter the hidden-layer matrix, the least-squares solves stay small, so training cost and prediction cost are lower than for a one-shot random layer of the same accuracy.
- The lower run-to-run dispersion reported for CD-DM implies the final model depends less on which random training points happen to be drawn.
- On the three real datasets, using two to five times fewer nodes makes cross-validated model selection cheaper and the deployed network smaller.
Reading between the lines
- The paper does not test the natural variant in which candidate hyperplanes are fitted to the current residual rather than to the original target; if that variant worked, the acceptance threshold would be a general-purpose residual-refinement rule rather than a filter specific to one generator.
- The threshold-halving schedule is a coarse-to-fine schedule: the construction suggests early accepted nodes encode the target's broad shape and later nodes encode detail, although the paper does not measure this decomposition directly.
- The same error-reduction acceptance rule should transplant to classification by substituting a classification loss, a direction the paper lists as future work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CD-DM, a constructive extension of the data-driven random hidden node method D-DM. The algorithm iteratively generates candidate sigmoid hidden nodes by sampling a training point x*, fitting a local hyperplane to the original target values in its neighborhood, and setting the node's weight vector and bias from that hyperplane. A candidate is accepted only if it decreases the training RMSE by more than an adaptive threshold; the threshold is halved whenever no node is accepted for Q consecutive attempts. Experiments compare CD-DM with D-DM on one univariate synthetic function, one bivariate synthetic function, and three real-world datasets, reporting medians and interquartile ranges over 100 runs. The authors claim faster convergence, more compact architectures, and, via a transitive comparison with an unpublished companion paper, superiority over standard and state-of-the-art randomized learning methods.
Significance. If the empirical claims hold, the paper makes a useful practical contribution to randomized feedforward learning: a data-dependent basis combined with greedy error-based selection can produce smaller networks than adding an entire random projection layer at once. The experimental protocol has notable strengths: 100 independent trials, separate test sets, median and IQR reporting, and cross-validation for the main hyperparameters m and k'. However, the central constructive mechanism is only partially validated, the state-of-the-art comparison rests on an unpublished anonymous baseline, and the accuracy differences are not significance-tested. The contribution is incremental rather than foundational, but it is of interest to researchers working on randomized neural networks and deserves further scrutiny.
major comments (3)
- [Algorithm 1, steps (a)-(e)] Candidate hidden nodes are generated from the original target values y, not from the current residual: step (c) fits the hyperplane to the neighborhood of x* using the original outputs, and step (k) is the only residual-dependent operation. The central compactness claim therefore assumes that the residual after previously accepted nodes can be approximated by further nodes drawn from the same target-derived family. This assumption is not argued for and is not tested; for example, no experiment generates candidates from the current residual y - Y'. If the residual has structure absent from the original local slopes, the procedure can only halve theta until weakly useful nodes pass the threshold, and the interpretation that accepted neurons are 'significant' would break down. I ask the authors to either supply a supporting argument or add an experimental variant with residual-based candidate generation; at minimum, this limitation should be stated explicitly.
- [Simulation study, final paragraph] The conclusion that CD-DM outperforms standard and state-of-the-art methods is a transitive argument based entirely on (Anonymous 2019a), an unpublished manuscript that the reader cannot access. The core node-generation equations (3)-(4) are also taken from that same source. To support the headline claim, the authors should either run published baselines directly on the four benchmark problems or restrict the claim to 'CD-DM is more compact and faster-converging than D-DM', with published references used only for qualitative context.
- [Simulation study, Table 1 and Fig. 4] The RMSE differences between CD-DM and D-DM are reported as medians with interquartile ranges, but no paired significance tests are provided. For Stock the intervals are 0.0285 +/- 0.0028 versus 0.0265 +/- 0.0014, and for Concrete 0.0770 +/- 0.0055 versus 0.0748 +/- 0.0034; these intervals overlap considerably. The compactness and convergence-speed claims are credible, but the accuracy advantage should be established by paired per-run tests, or the paper should state that no consistent accuracy improvement is claimed.
minor comments (4)
- [Algorithm 1] The initialization RMSE0 = 1 assumes the target outputs have been normalized to [0,1]; this normalization should be stated explicitly before the algorithm is presented.
- [Abstract and Constructive Approach section] The text says the threshold theta is 'reduced by half', but because theta is initialized to a negative value, the operation theta = theta/2 increases it toward zero. The sign convention should be stated explicitly to avoid confusion.
- [Simulation study] The phrase 'faster convergence' is always measured in terms of the number of hidden nodes; no wall-clock training times are reported. The authors should state that the advantage is in hidden-node count and not claim a runtime advantage without timing experiments.
- [Related work] The discussion of constructive randomized networks is thin given that construction is central to the paper; citing and comparing with incremental extreme learning machines and stochastic configuration networks would help position the contribution.
Circularity Check
CD-DM vs D-DM is independently tested, but the node-generation formulas and the state-of-the-art conclusion are imported from the author's own unpublished anonymous manuscript.
-
self citation load bearing
[Constructive Approach for Data-Driven Randomized Learning, Eqs. (3)-(4); Conclusion]
"we obtain the sigmoid weights (see (Anonymous 2019a) for details): aj = 4a′ j, j = 1, 2, ..., n (3) ... b = − aT x∗ (4) ... Taking into account the performance comparison reported in (Anonymous 2019a), where the original D-DM are compared with the standard methods ... we can conclude that CD-DM outperforms standard and state-of-the-art methods in terms of accuracy, convergence speed and more compact network architecture."
The core formulas that define the data-driven candidate generation are not derived in this paper; they are deferred to an unpublished, same-author manuscript (Anonymous 2019a). Likewise, the paper's only route from its own experiments (CD-DM vs D-DM only) to the claim that CD-DM outperforms standard and state-of-the-art methods is the comparison reported in that same self-cited manuscript. Thus two load-bearing links in the argument are supplied by self-citation that the reader cannot verify. This is not full circularity because CD-DM's advantage over D-DM is evaluated on held-out data and is not constructed from the conclusion.
full rationale
The central comparison in the paper—CD-DM reaching the same error as D-DM with 33 nodes versus 250 nodes on the single-variable function, and using two-to-five-fold fewer nodes on the real datasets—is produced by running both algorithms and measuring held-out RMSE; this is independent evidence and is not forced by the definitions. The acceptance rule does make 'an accepted node reduces training error' true by construction, but the quantitative compactness claim and the lower test RMSE are empirical. The main weakness is that the hidden-node construction (Eqs. 3-4) and the state-of-the-art superiority claim are sourced to Anonymous 2019a, an unpublished same-author manuscript, rather than derived or benchmarked in this paper. That is a load-bearing self-citation chain for those parts, though not for the core CD-DM-vs-D-DM result. The phrase 'includes only significant neurons' is definitional rather than a separate prediction, so it is not counted as an additional circular step. Score 3 reflects partial self-citation, not construction-level circularity.
Assumptions & free parameters
free parameters (4)
- final number of hidden nodes m =
160 (Function 7), 110 (Stock), 50 (Concrete), 120 (Compactiv)
- neighborhood size k' =
35 (Function 7), 30 (Stock), 8 (Concrete), 25 (Compactiv)
- initial acceptance threshold θ =
-0.01
- patience Q (iterations without accepted node before halving θ) =
50
assumptions (4)
- standard math Randomized single-hidden-layer networks with sigmoids satisfy universal approximation conditions from the literature (Husmeier 1999).
- ad hoc to paper The local hyperplane fitted to the k nearest neighbors of x* approximates the target function slope well enough to set sigmoid slopes via a_j = 4 a'_j.
- domain assumption Drawing x* uniformly from the training set and anchoring the sigmoid inflection point at x* covers the input space adequately.
- standard math The Moore-Penrose pseudoinverse gives the optimal output weights for the frozen random hidden layer.
Cite this review
Pith. "Pith review of A Constructive Approach for Data-Driven Randomized Learning of Feedforward Neural Networks." pith.science (2026). https://pith.science/paper/XOMAB7XQ
@misc{pith2026190901961,
author = {Pith},
title = {Pith review of: A Constructive Approach for Data-Driven Randomized Learning of Feedforward Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/XOMAB7XQ}},
note = {Machine review of arXiv:1909.01961}
}
read the original abstract
Feedforward neural networks with random hidden nodes suffer from a problem with the generation of random weights and biases as these are difficult to set optimally to obtain a good projection space. Typically, random parameters are drawn from an interval which is fixed before or adapted during the learning process. Due to the different functions of the weights and biases, selecting them both from the same interval is not a good idea. Recently more sophisticated methods of random parameters generation have been developed, such as the data-driven method proposed in \cite{Anon19}, where the sigmoids are placed in randomly selected regions of the input space and then their slopes are adjusted to the local fluctuations of the target function. In this work, we propose an extended version of this method, which constructs iteratively the network architecture. This method successively generates new hidden nodes and accepts them if the training error decreases significantly. The threshold of acceptance is adapted to the current training stage. At the beginning of the training process only those nodes which lead to the largest error reduction are accepted. Then, the threshold is reduced by half to accept those nodes which model the target function details more accurately. This leads to faster convergence and more compact network architecture, as it includes only "significant" neurons. Several application examples are given which confirm this thesis.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Anonymous. 2019a. Data-driven randomized learning of feedforward neural networks. (in review)
-
[2]
Anonymous. 2019b. Generating random weights and biases in feedforward neural networks with random hidden nodes. Information Sciences 481:33--56
-
[3]
Anonymous. 2019c. Improving randomized learning of feedforward neural networks by appropriate generation of random parameters. In Rojas, I.; Joya, G.; and Catala, A., eds., Advances in Computational Intelligence , 517--530. Cham: Springer International Publishing
-
[4]
Cao, W.; Wang, X.; Ming, Z.; and Gao, J. 2018. A review on neural networks with random weights. Neurocomputing 275:278--287
work page 2018
-
[5]
Dai, W.; Li, D.; Zhou, P.; and Chai, T. 2019. Stochastic configuration networks with block increments for data modeling in process industries. Information Sciences 484:367 -- 386
work page 2019
-
[6]
Husmeier, D. 1999. Random vector functional link (rvfl) networks. In Neural Networks for Conditional Probability Estimation: Forecasting Beyond Point Predictions . Springer-Verlag London. chapter 6, 87--97
work page 1999
-
[7]
Li, M., and Wang, D. 2017. Insights into randomized algorithms for neural networks: Practical issues and common pitfalls. Information Sciences 382--383:170--178
work page 2017
-
[8]
Pao, Y.; Park, G.; and Sobajic, D. 1994. Learning and generalization characteristics of the random vector functional-link net. Neurocomputing 6(2):163--180
work page 1994
Show all 12 references
-
[9]
Principe, J., and Chen, B. 2015. Universal approximation with convex optimization: Gimmick or reality? IEEE Comput Intell Mag 10:68--77
2015
-
[10]
Wang, D., and Li, M. 2017. Stochastic configuration networks: Fundamentals and algorithms. IEEE Trans. Cybernetics 47(10):3466--3479
2017
-
[11]
Zhang, L., and Suganthan, P. 2016. A survey of randomized algorithms for training neural networks. Information Sciences 364:146--155
2016
-
[12]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.