REVIEW 4 major objections 6 minor 15 references
Effective Non-Random Extreme Learning Machine
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper proposes replacing random ELM hidden weights with a data-derived eigenbasis of the NNGP kernel, yielding two deterministic variants with comparable regression accuracy and cheaper model selection.
desk verdict A clever deterministic-ELM construction with reproducible experiments, but the derivation of the optimal weights is mathematically off and the test-set model selection inflates the results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the spectral decomposition of the NNGP kernel Gram matrix: for a shallow random network in the infinite-width limit, $K_\sigma$ is the expected Gram matrix of the hidden features, and its eigenvector matrix $U$ supplies an orthonormal basis for that feature space. The paper's key identity is the closed-form minimizer $\hat{W} = \sigma^{-1}(U^\top) X^\top (X X^\top)^{-1}$ of $\min_W \|\sigma(W X)^\top - U\|_F$, which converts the target basis into hidden-layer weights. A-ENR-ELM uses $U$ directly as a proxy for the hidden output matrix, while I-ENR-ELM uses the realized hidden output $S$ in an incremental forward stagewise fit. The neuron-ordering mechanism is what makes both variants produce a full error curve at low cost, and that error curve is the basis for architecture selection.
What would settle it
Compute, on a small-input-dimension regression problem, the normalized Frobenius error $\|\sigma(\hat{W}X)^\top - U\|_F/\|U\|_F$ along with the gap between A-ENR-ELM's test error and the oracle test error of regressing $y$ directly on $U$; if both gaps are large while the method still matches ELM's accuracy, the optimality claim for $\hat{W}$ is not what carries the performance.
Extended reading notes
Core claim
The central claim is that a shallow ELM's random hidden layer can be replaced by a data-dependent one without losing predictive accuracy, and that the replacement is optimal in a least-squares sense. Given the NNGP kernel matrix $K_\sigma = (K_\sigma(x_i, x_j))_{i,j=1,\dots,T}$ of the training inputs, the paper takes its spectral decomposition $K_\sigma = U \Delta U^\top$ and orders the columns of $U$ by decreasing $|\langle y, U_{\cdot j}\rangle|$. The first $n$ columns then form an orthonormal basis for the most informative $n$-dimensional subspace of the infinite-width feature space. Since no exact feature map realizes that basis, the paper solves $\min_W \|\sigma(W X)^\top - U\|_F$ to obtain $\hat{W} = \sigma^{-1}(U^\top) X^\top (X X^\top)^{-1}$, where $\sigma$ is an invertible activation with range $[-1,1]$, taken as erf in the experiments. A-ENR-ELM then regresses $y$ on $U$ itself with $\hat{\beta} = (U_{\cdot J})^\top y$, while I-ENR-ELM runs incremental forward stagewise regression on $S = \sigma(\hat{W} X)^\top$; both variants return a nested family of models, so one test-error curve can be evaluated and used for model selection.
Load-bearing premise
The method's load-bearing premise is that the kernel's eigenvectors can be well approximated by the hidden layer's fitted outputs, even though the construction guarantees no bound on the approximation error and the authors note that the exact oracle does not exist.
Editorial extensions
If this is right
- Model selection for a single-layer regressor can be read off a test-error curve computed from one nested family of models, rather than by retraining many random realizations for each candidate number of hidden neurons.
- Hidden-layer weights are fixed by the training inputs and the activation function, so two runs of the method on the same data produce the same model, removing the random-initialization sensitivity of standard ELM.
- Output-layer coefficients cost linear time in the A-ENR-ELM variant and incremental forward stagewise steps in the I-ENR-ELM variant, with no matrix inversion, which the paper reports as a large reduction in model-selection time on small and medium datasets.
- Both variants' training and test error curves saturate instead of interpolating, which the paper interprets as resistance to the overfitting exhibited by the traditional ELM baseline.
- The construction requires an invertible activation with range $[-1,1]$, and the experiments use the erf activation throughout.
Reading between the lines
- Because the construction only needs a positive-semidefinite Gram matrix of data-dependent features, the same two-phase scheme could be tried with kernels other than the NNGP kernel; the paper does not test this.
- The rank mismatch between $U$, which can have up to $T$ eigenvectors, and $\sigma(\hat{W} X)^\top$, whose rank is at most the input dimension $n_0$, suggests a measurable diagnostic: track the normalized Frobenius distance between the two matrices across datasets and see whether it predicts when A-ENR-ELM and I-ENR-ELM diverge.
- For large training sets, the eigendecomposition of the $T \times T$ Gram matrix dominates cost, so subsampling the training set to construct the basis is a natural extension that the paper mentions only as a possibility.
- The nested-family error-curve idea could carry over to classification by swapping the squared loss for a class-separability loss, which the paper notes but does not implement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a deterministic alternative to the random hidden-layer weights of Extreme Learning Machines (ELMs) for regression. The method, ENR-ELM, computes the NNGP kernel Gram matrix of the training inputs, takes its eigenvector matrix U, and then attempts to find hidden weights W such that the hidden features sigma(W X)^T approximate U. Two variants are given: A-ENR-ELM, which fits the output layer directly on U, and I-ENR-ELM, which uses incremental forward-stagewise regression on the actual hidden features. The authors report comparable predictive accuracy to classic ELM on synthetic and real datasets while greatly reducing the computational cost of model selection.
Significance. If the central derivation were correct, the paper would offer a principled way to remove randomness from ELM training and to build error curves for model selection in a single pass. The empirical study is broad, spanning 48 synthetic configurations and 8 real datasets, and the authors provide reproducible code and detailed timing comparisons, which are valuable strengths. However, the theoretical justification of the proposed 'optimal' weight matrix is not sound, as detailed below; the method may still work as a heuristic, but the paper's central claim is currently unsupported.
major comments (4)
- [Section 3, Eq. (9)] The closed-form solution W_hat = σ^{-1}(U^T) X^T (X X^T)^{-1} does not solve the optimization problem in Eq. (9). The derivation applies the inverse activation to the target before projecting, which minimizes ||σ^{-1}(U^T) - W X||_F, not ||σ(W X)^T - U||_F. Because σ is nonlinear, the Frobenius norm is not preserved under applying σ^{-1}, so the two objectives are different. The manuscript provides no bound on the actual objective ||σ(W_hat X)^T - U||_F. Since the 'optimal weights matrix' claim is the theoretical basis for the method, this is a load-bearing gap.
- [Section 3, after Eq. (9)] Even if a different objective were intended, W_hat X has rank at most n0, whereas U is generically full rank with T columns (or at least rank T for the selected n columns). When n0 < T, exact or near-exact matching is impossible, yet no approximation error is quantified. Section 3's admission that 'the oracle does not exist' underscores this, but the paper still calls W_hat optimal; a formal error bound or a clear statement of heuristic status is needed.
- [Section 3, Algorithm 2 and Section 4.2] The A-ENR-ELM variant compounds the previous issue: Algorithm 2 computes β by regressing y on U, but the prediction at test time uses σ(W_hat x), i.e., the actual hidden features. Since σ(W_hat X)^T is not equal to U, the training and test feature maps are different objects. This train/test mismatch is not analyzed; it may explain the method's numerical behavior, but it invalidates the interpretation of β as the OLS solution for the actual features used at prediction time.
- [Section 3, Eq. (9) and Algorithms 1-3] There is a dimensional inconsistency: the minimization in Eq. (9) is stated over W ∈ R^{T×n0}, whereas a hidden weight matrix should be in R^{n×n0}; the closed-form formula produces a T×n0 matrix. After the column selection in Algorithms 2 and 3, W_hat remains T×|J|, not n×n0. The dimensions of all matrices in Eq. (9) and the algorithms should be stated consistently.
minor comments (6)
- [Section 2.3] The acronym 'NNPG' appears in the third paragraph and should be 'NNGP'.
- [Section 3] The phrase 'non-negative defined' should be 'non-negative definite'.
- [Throughout] Some table and figure labels contain a stray space, e.g., 'T able 1' and 'T able 3'; these should be fixed.
- [Section 4.2 and Figure 2] The 'greatest curvature' model-selection criterion is mentioned but never defined operationally; specify how curvature is computed (e.g., second differences of the test error curve) and justify its use.
- [Section 4 and Tables 3, 6] The reported test errors are minima of test-error curves used for model selection; this is an optimistic selection procedure. The authors should state that these numbers are not unbiased generalization estimates, though the comparison between methods is on equal footing.
- [Algorithm 3] The stopping rule uses a tolerance toll, but the choice of toll is not discussed; a brief sensitivity note would help.
Circularity Check
No significant circularity: the NNGP kernel provides an external target, and the final-layer fits are ordinary least-squares/projection steps; the Eq. (9) closed-form gap is a correctness issue, not a circular reduction.
full rationale
K_sigma is computed from the external NNGP kernel package [13] (or the standard recursion in Section 2.3), U is its eigen-decomposition, and the output weights are obtained by orthogonal projection beta = U_J^T y (A-ENR-ELM) or by incremental forward-stagewise least squares on S = sigma(W_hat X)^T (I-ENR-ELM). No parameter is defined as the quantity it later predicts: y enters only through supervised column ordering and through the final least-squares fit, which is standard supervised learning, and the reported test errors are evaluated on held-out data. The single author-overlapping citation, [12], appears only in a background list of NNGP convergence theory ('the theoretical understanding of this topic is quite extensive, see e.g. [10], [11], [12]') and is not used to derive the construction. The paper itself concedes 'The oracle does not exist' and supplies no bound on ||sigma(W_hat X)^T - U||_F, and the closed form W_hat = sigma^{-1}(U^T) X^T (XX^T)^{-1} is not actually the minimizer of the stated objective (9) for nonlinear sigma; those are correctness/derivation gaps, not circularity, because W_hat is fitted to an external kernel object rather than to the response, and the test predictions are not forced by construction to equal any fitted value. Hence no load-bearing circular step is present.
Assumptions & free parameters
free parameters (3)
- Number of hidden neurons n =
selected from test error curve (up to min(50*n0, T/2))
- epsilon (I-ENR-ELM step size) =
not reported in the experiments
- tol (I-ENR-ELM stopping tolerance) =
not reported
assumptions (5)
- standard math The NNGP kernel formula with erf dual activation is valid and computable via the method of Han et al. [13].
- domain assumption The activation function is invertible and maps R to [-1,1], satisfied by erf.
- ad hoc to paper The target eigenbasis U can be approximately realized as sigma(W X)^T despite rank(W X) <= n0 < T.
- domain assumption The training matrix X has full row rank so XX^T is invertible.
- ad hoc to paper The test error curve is a valid guide for choosing n and for reporting performance.
Cite this review
Pith. "Pith review of Effective Non-Random Extreme Learning Machine." pith.science (2026). https://pith.science/paper/COOO7FIP
@misc{pith2026241116229,
author = {Pith},
title = {Pith review of: Effective Non-Random Extreme Learning Machine},
year = {2026},
howpublished = {\url{https://pith.science/paper/COOO7FIP}},
note = {Machine review of arXiv:2411.16229}
}
read the original abstract
The Extreme Learning Machine (ELM) is a growing statistical technique widely applied to regression problems. In essence, ELMs are single-layer neural networks where the hidden layer weights are randomly sampled from a specific distribution, while the output layer weights are learned from the data. Two of the key challenges with this approach are the architecture design, specifically determining the optimal number of neurons in the hidden layer, and the method's sensitivity to the random initialization of hidden layer weights. This paper introduces a new and enhanced learning algorithm for regression tasks, the Effective Non-Random ELM (ENR-ELM), which simplifies the architecture design and eliminates the need for random hidden layer weight selection. The proposed method incorporates concepts from signal processing, such as basis functions and projections, into the ELM framework. We introduce two versions of the ENR-ELM: the approximated ENR-ELM and the incremental ENR-ELM. Experimental results on both synthetic and real datasets demonstrate that our method overcomes the problems of traditional ELM while maintaining comparable predictive performance.
Reference graph
Works this paper leans on
-
[1]
Scardapane, S., Wang, D.: Randomness in neural networks: an overview. WIREs Data Mining and Knowledge Dis- covery 7(2), 1200 (2017) https://doi.org/10.1002/widm.1200 https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/widm.1200
-
[2]
Patil, H., Sharma, K.: Extreme learning machine: A comprehensive survey of theories and algorithms. In: 2023 International Conference on Computational Intelligence and Sustainable Engineering Solutions (CISES), pp. 749–756 (2023). https://doi.org/10.1109/CISES58720.2023.10183613
-
[3]
Neural Comput and Applic 33, 15121–15144 (2021) https://doi.org/10.1007/s00521-021-06402-y
Markowska-Kaczmar, U., Kosturek, M.: Extreme learning machine versus clas- sical feedforward network. Neural Comput and Applic 33, 15121–15144 (2021) https://doi.org/10.1007/s00521-021-06402-y
-
[5]
Neurocomputing 74(16), 2483–2490 (2011) https://doi.org/10.1016/j
Wang, Y., Cao, F., Yuan, Y.: A study on effectiveness of extreme learning machine. Neurocomputing 74(16), 2483–2490 (2011) https://doi.org/10.1016/j. neucom.2010.11.030 . Advances in Extreme Learning Machine: Theory and Appli- cations Biological Inspired Systems. Computational and Ambient Intelligence
doi:10.1016/j 2011
-
[6]
Giryes, R., Sapiro, G., Bronstein, A.M.: Deep neural networks with random gaus- sian weights: A universal classification strategy? IEEE Transactions on Signal Processing 64(13), 3444–3457 (2016) https://doi.org/10.1109/TSP.2016.2546221
arXiv 2016
-
[7]
IEEE Transactions on Cybernetics 47(10), 3466–3479 (2017) https://doi
Wang, D., Li, M.: Stochastic configuration networks: Fundamentals and algo- rithms. IEEE Transactions on Cybernetics 47(10), 3466–3479 (2017) https://doi. org/10.1109/TCYB.2017.2734043
arXiv 2017
-
[8]
Information Sciences 185(1), 66–77 (2012) https://doi.org/10.1016/j.ins.2011.09
Cao, J., Lin, Z., Huang, G.-B., Liu, N.: Voting based extreme learning machine. Information Sciences 185(1), 66–77 (2012) https://doi.org/10.1016/j.ins.2011.09. 015
-
[9]
Neal, R.M.: Priors for Infinite Networks, pp. 29–53. Springer, New York, NY (1996). https://doi.org/10.1007/978-1-4612-0745-0 2 . https://doi.org/10.1007/ 978-1-4612-0745-0 2
Show all 15 references
-
[10]
The Annals of Applied Probability 33(6A), 4798–4819 (2023) https: //doi.org/10.1214/23-AAP1933
Hanin, B.: Random neural networks in the infinite width limit as Gaussian processes. The Annals of Applied Probability 33(6A), 4798–4819 (2023) https: //doi.org/10.1214/23-AAP1933
2023 doi
- [11]
-
[12]
Stochastic Systems 0(0), (0) https://doi.org/10.1287/stsy.2023.0033
Apollonio, N., De Canditiis, D., Franzina, G., Stolfi, P., Torrisi, G.L.: Normal approximation of random gaussian neural networks. Stochastic Systems 0(0), (0) https://doi.org/10.1287/stsy.2023.0033
2023
-
[13]
In: Proceedings of the 36th International Con- ference on Neural Information Processing Systems
Han, I., Zandieh, A., Lee, J., Novak, R., Xiao, L., Karbasi, A.: Fast neural kernel embeddings for general activations. In: Proceedings of the 36th International Con- ference on Neural Information Processing Systems. NIPS ’22. Curran Associates Inc., Red Hook, NY, USA (2024)
2024
-
[14]
Pattern Recognition 96, 106960 (2019) https: //doi.org/10.1016/j.patcog.2019.07.005
Zhang, W., Zhang, Z., Wang, L., Chao, H.-C., Zhou, Z.: Extreme learning machines with expectation kernels. Pattern Recognition 96, 106960 (2019) https: //doi.org/10.1016/j.patcog.2019.07.005
2019 doi
-
[15]
Springer series in statistics
Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer series in statistics. Springer, ??? (2009). https://books.google.it/books?id=eBSgoAEACAAJ
2009
-
[16]
https://archive.ics.uci.edu 33
Kelly, M., Longjohn, R., Nottingham, K.: The UCI Machine Learning Repository. https://archive.ics.uci.edu 33
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.