{"id":"1b21e845-934e-4f58-afb6-1ae440830364","arxiv_id":"2411.16229","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ENR-ELM constructs deterministic hidden-layer features from the eigenvectors of the NNGP kernel and fits the output layer by projection or incremental regression, matching ELM accuracy with less randomness and faster model selection.","lead":"The authors propose a regression method, Effective Non-Random ELM, that replaces random hidden-layer weights in Extreme Learning Machines with data-dependent weights built from the kernel Gram matrix of an infinite-width network. It offers a deterministic alternative that avoids random initialization and speeds up model selection while matching ELM accuracy on the tested datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The closed-form W_hat in Section 3 does not minimize Eq. (9): inverting sigma before projecting onto the row space of X solves a different objective, so the claimed optimality of the data-dependent weights is unsupported.","rationale":"The reader's weakest assumption identifies the rank/approximation gap between U and sigma(W_hat X)^T, which is real. The more direct and more easily checkable failure is that the claimed closed form is not a minimizer of the stated objective: applying sigma^{-1} before the row-space projection changes the objective unless sigma is linear. This is an internal mathematical inconsistency, not merely an unquantified approximation. I do not push to rejection because the empirical study is broad, the code is public, and the method may still be useful as a deterministic heuristic if the 'optimal' claim is removed or replaced with an explicit approximation guarantee. The reader's CONDITIONAL verdict remains appropriate, but the conditions should include reworking or softening the W_hat derivation and, ideally, a direct numerical check of the objective in Eq. (9).","tokens_in":18512,"tokens_out":8213,"duration_ms":83601,"concrete_test":"On a small synthetic problem (for example, T=100, n0=10, erf activation, standardized X), compute K_sigma, U, and the closed-form W_hat. Using automatic differentiation, compute the gradient of F(W) = ||sigma(WX)^T - U||_F^2 at W_hat. If the gradient is nonzero, or if a small number of gradient-descent steps decreases F below F(W_hat), then W_hat is not the argmin of Eq. (9); report the relative decrease and the final residual to settle the optimality claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines W_hat as argmin_W ||sigma(WX)^T - U||_F, then claims the closed-form solution is W_hat = sigma^{-1}(U^T) X^T (X X^T)^{-1}. This is not a solution of Eq. (9). The closed form instead minimizes ||sigma^{-1}(U^T) - W X||_F: it applies the inverse activation to the target and then projects onto the row space of X. Because sigma is nonlinear, minimizing distance in the pre-image domain is not equivalent to minimizing ||sigma(WX)^T - U||_F; the Frobenius norm is not preserved under sigma. No bound on ||sigma(W_hat X)^T - U||_F is provided, and W_hat X has rank at most n0 while sigma^{-1}(U^T) is generically full rank, so the residual can be large. A-ENR-ELM further compounds this by fitting beta on U but predicting with sigma(W_hat x), so the training and test feature maps are not the same object. The 'optimal weights matrix' assertion is therefore not supported by the derivation, even though the empirical comparisons may still indicate the construction works as a heuristic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deterministic alternative to the random hidden-layer weights of Extreme Learning Machines (ELMs) for regression. The method, ENR-ELM, computes the NNGP kernel Gram matrix of the training inputs, takes its eigenvector matrix U, and then attempts to find hidden weights W such that the hidden features sigma(W X)^T approximate U. Two variants are given: A-ENR-ELM, which fits the output layer directly on U, and I-ENR-ELM, which uses incremental forward-stagewise regression on the actual hidden features. The authors report comparable predictive accuracy to classic ELM on synthetic and real datasets while greatly reducing the computational cost of model selection.","tokens_in":18820,"tokens_out":4922,"duration_ms":43259,"significance":"If the central derivation were correct, the paper would offer a principled way to remove randomness from ELM training and to build error curves for model selection in a single pass. The empirical study is broad, spanning 48 synthetic configurations and 8 real datasets, and the authors provide reproducible code and detailed timing comparisons, which are valuable strengths. However, the theoretical justification of the proposed 'optimal' weight matrix is not sound, as detailed below; the method may still work as a heuristic, but the paper's central claim is currently unsupported.","major_comments":[{"comment":"The closed-form solution W_hat = σ^{-1}(U^T) X^T (X X^T)^{-1} does not solve the optimization problem in Eq. (9). The derivation applies the inverse activation to the target before projecting, which minimizes ||σ^{-1}(U^T) - W X||_F, not ||σ(W X)^T - U||_F. Because σ is nonlinear, the Frobenius norm is not preserved under applying σ^{-1}, so the two objectives are different. The manuscript provides no bound on the actual objective ||σ(W_hat X)^T - U||_F. Since the 'optimal weights matrix' claim is the theoretical basis for the method, this is a load-bearing gap.","section":"Section 3, Eq. (9)"},{"comment":"Even if a different objective were intended, W_hat X has rank at most n0, whereas U is generically full rank with T columns (or at least rank T for the selected n columns). When n0 < T, exact or near-exact matching is impossible, yet no approximation error is quantified. Section 3's admission that 'the oracle does not exist' underscores this, but the paper still calls W_hat optimal; a formal error bound or a clear statement of heuristic status is needed.","section":"Section 3, after Eq. (9)"},{"comment":"The A-ENR-ELM variant compounds the previous issue: Algorithm 2 computes β by regressing y on U, but the prediction at test time uses σ(W_hat x), i.e., the actual hidden features. Since σ(W_hat X)^T is not equal to U, the training and test feature maps are different objects. This train/test mismatch is not analyzed; it may explain the method's numerical behavior, but it invalidates the interpretation of β as the OLS solution for the actual features used at prediction time.","section":"Section 3, Algorithm 2 and Section 4.2"},{"comment":"There is a dimensional inconsistency: the minimization in Eq. (9) is stated over W ∈ R^{T×n0}, whereas a hidden weight matrix should be in R^{n×n0}; the closed-form formula produces a T×n0 matrix. After the column selection in Algorithms 2 and 3, W_hat remains T×|J|, not n×n0. The dimensions of all matrices in Eq. (9) and the algorithms should be stated consistently.","section":"Section 3, Eq. (9) and Algorithms 1-3"}],"minor_comments":[{"comment":"The acronym 'NNPG' appears in the third paragraph and should be 'NNGP'.","section":"Section 2.3"},{"comment":"The phrase 'non-negative defined' should be 'non-negative definite'.","section":"Section 3"},{"comment":"Some table and figure labels contain a stray space, e.g., 'T able 1' and 'T able 3'; these should be fixed.","section":"Throughout"},{"comment":"The 'greatest curvature' model-selection criterion is mentioned but never defined operationally; specify how curvature is computed (e.g., second differences of the test error curve) and justify its use.","section":"Section 4.2 and Figure 2"},{"comment":"The reported test errors are minima of test-error curves used for model selection; this is an optimistic selection procedure. The authors should state that these numbers are not unbiased generalization estimates, though the comparison between methods is on equal footing.","section":"Section 4 and Tables 3, 6"},{"comment":"The stopping rule uses a tolerance toll, but the choice of toll is not discussed; a brief sensitivity note would help.","section":"Algorithm 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical component and reproducible code, but the central theoretical claim in Section 3 is not supported. The authors may be able to revise by reframing the method as a heuristic and either providing approximation error bounds or discussing the mismatch more honestly; in that case the empirical contribution could be publishable. The dimensional issue in Eq. (9) should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fabio, this one is worth reading but not for the reason the authors think. The construction is genuinely new: take the NNGP kernel of the input data, take its top eigenvectors U, invert the activation, and project back onto the data to get hidden weights. That gives you a deterministic ELM with no random draw, and the incremental variant gives you an error curve in one pass. That is a real contribution to the ELM toolkit. The code is public, the experiments are broad (48 synthetic and 8 real datasets), and the computational time comparison is credible—they compare full model selection for both methods, not a cherry-picked single run.\n\nThe soft spot is in Section 3, and it is load-bearing. The problem they state is W_hat = argmin_W ||sigma(WX)^T - U||_F. The closed form they give, sigma^{-1}(U^T) X^T (XX^T)^{-1}, solves a different problem: minimize ||sigma^{-1}(U^T) - WX||_F. Nonlinear sigma doesn't commute with projection, so there is no reason this minimizes the stated objective. On top of that, WX has rank at most n0, while U is full rank T, so exact matching is impossible. They say 'the oracle does not exist' but then still call W_hat optimal. That needs either a bound on the approximation error or a rewriting of the paper as a heuristic.\n\nThe second issue is evaluation. They use the test error curve to select the number of neurons and then report the minimum of that same curve. That is selection on the test set, and the reported numbers are optimistic. A clean protocol would be validation-set selection and test-set reporting.\n\nA-ENR-ELM also has a train/test mismatch: beta is fit on U, but prediction uses sigma(W_hat x). The paper is aware it is an approximation, but the bias is unquantified.\n\nMinor: epsilon and tol for I-ENR-ELM are left as free parameters with no guidance.\n\nBottom line: the core mathematical claim is not supported, and the evaluation protocol needs fixing. But the idea is interesting enough, and the empirical behavior is consistent enough across many datasets, that it deserves a serious referee. If I were editing, I'd send it out with a request to fix the derivation (or reposition the method) and redo the model-selection protocol.","headline":"A clever deterministic-ELM construction with reproducible experiments, but the derivation of the optimal weights is mathematically off and the test-set model selection inflates the results.","tokens_in":19315,"tokens_out":3752,"would_cite":true,"duration_ms":196558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes replacing random ELM hidden weights with a data-derived eigenbasis of the NNGP kernel, yielding two deterministic variants with comparable regression accuracy and cheaper model selection.","keywords":["Extreme Learning Machine","NNGP kernel","random features","non-parametric regression","model selection","incremental forward stagewise regression","spectral decomposition","data-dependent hidden weights"],"falsifier":"Compute, on a small-input-dimension regression problem, the normalized Frobenius error $\\|\\sigma(\\hat{W}X)^\\top - U\\|_F/\\|U\\|_F$ along with the gap between A-ENR-ELM's test error and the oracle test error of regressing $y$ directly on $U$; if both gaps are large while the method still matches ELM's accuracy, the optimality claim for $\\hat{W}$ is not what carries the performance.","tokens_in":18291,"feed_emoji":"⚡","tokens_out":8554,"duration_ms":74769,"temperature":0.7,"pith_summary":"Extreme Learning Machines fit a linear output layer on top of hidden features whose weights are drawn at random, leaving two practical problems: how many hidden neurons to use and which random draw to trust. The paper introduces the Effective Non-Random ELM, which fixes the hidden-layer weights from data instead of sampling them. It takes the eigenbasis $U$ of the NNGP kernel Gram matrix, the expected feature Gram matrix of an infinitely wide random hidden layer, solves in closed form for hidden weights $\\hat{W}$ whose outputs approximate that basis, and then fits the output layer either directly on $U$ or by incremental forward stagewise regression on the actual hidden outputs. On synthetic and real regression datasets, the paper reports test error comparable to standard ELM while cutting model-selection time and avoiding the overfitting seen in the random baseline. If these results hold, deterministic training and error-curve-based architecture selection become practical for small to medium regression problems.","feed_headline":"Non-random hidden weights match ELM accuracy at lower cost","feed_subtitle":"Replacing random hidden weights with a data-derived eigenbasis makes Extreme Learning Machines reproducible and quicker to tune.","key_machinery":"The machinery is the spectral decomposition of the NNGP kernel Gram matrix: for a shallow random network in the infinite-width limit, $K_\\sigma$ is the expected Gram matrix of the hidden features, and its eigenvector matrix $U$ supplies an orthonormal basis for that feature space. The paper's key identity is the closed-form minimizer $\\hat{W} = \\sigma^{-1}(U^\\top) X^\\top (X X^\\top)^{-1}$ of $\\min_W \\|\\sigma(W X)^\\top - U\\|_F$, which converts the target basis into hidden-layer weights. A-ENR-ELM uses $U$ directly as a proxy for the hidden output matrix, while I-ENR-ELM uses the realized hidden output $S$ in an incremental forward stagewise fit. The neuron-ordering mechanism is what makes both variants produce a full error curve at low cost, and that error curve is the basis for architecture selection.","core_discovery":"The central claim is that a shallow ELM's random hidden layer can be replaced by a data-dependent one without losing predictive accuracy, and that the replacement is optimal in a least-squares sense. Given the NNGP kernel matrix $K_\\sigma = (K_\\sigma(x_i, x_j))_{i,j=1,\\dots,T}$ of the training inputs, the paper takes its spectral decomposition $K_\\sigma = U \\Delta U^\\top$ and orders the columns of $U$ by decreasing $|\\langle y, U_{\\cdot j}\\rangle|$. The first $n$ columns then form an orthonormal basis for the most informative $n$-dimensional subspace of the infinite-width feature space. Since no exact feature map realizes that basis, the paper solves $\\min_W \\|\\sigma(W X)^\\top - U\\|_F$ to obtain $\\hat{W} = \\sigma^{-1}(U^\\top) X^\\top (X X^\\top)^{-1}$, where $\\sigma$ is an invertible activation with range $[-1,1]$, taken as erf in the experiments. A-ENR-ELM then regresses $y$ on $U$ itself with $\\hat{\\beta} = (U_{\\cdot J})^\\top y$, while I-ENR-ELM runs incremental forward stagewise regression on $S = \\sigma(\\hat{W} X)^\\top$; both variants return a nested family of models, so one test-error curve can be evaluated and used for model selection.","pith_inferences":["Because the construction only needs a positive-semidefinite Gram matrix of data-dependent features, the same two-phase scheme could be tried with kernels other than the NNGP kernel; the paper does not test this.","The rank mismatch between $U$, which can have up to $T$ eigenvectors, and $\\sigma(\\hat{W} X)^\\top$, whose rank is at most the input dimension $n_0$, suggests a measurable diagnostic: track the normalized Frobenius distance between the two matrices across datasets and see whether it predicts when A-ENR-ELM and I-ENR-ELM diverge.","For large training sets, the eigendecomposition of the $T \\times T$ Gram matrix dominates cost, so subsampling the training set to construct the basis is a natural extension that the paper mentions only as a possibility.","The nested-family error-curve idea could carry over to classification by swapping the squared loss for a class-separability loss, which the paper notes but does not implement."],"forward_implications":["Model selection for a single-layer regressor can be read off a test-error curve computed from one nested family of models, rather than by retraining many random realizations for each candidate number of hidden neurons.","Hidden-layer weights are fixed by the training inputs and the activation function, so two runs of the method on the same data produce the same model, removing the random-initialization sensitivity of standard ELM.","Output-layer coefficients cost linear time in the A-ENR-ELM variant and incremental forward stagewise steps in the I-ENR-ELM variant, with no matrix inversion, which the paper reports as a large reduction in model-selection time on small and medium datasets.","Both variants' training and test error curves saturate instead of interpolating, which the paper interprets as resistance to the overfitting exhibited by the traditional ELM baseline.","The construction requires an invertible activation with range $[-1,1]$, and the experiments use the erf activation throughout."],"supporting_citations":[{"why":"Supplies the fast algorithm that evaluates the NNGP kernel matrix $K_\\sigma$ whose eigenbasis defines $U$.","marker":"[13]"},{"why":"Establishes the ELM-to-kernel correspondence that motivates using the NNGP kernel to fix the hidden layer.","marker":"[14]"},{"why":"Is the prior data-dependent hidden-weight construction the paper seeks to generalize beyond radial basis functions.","marker":"[5]"},{"why":"Provides the incremental constructive strategy that I-ENR-ELM adapts to build the nested family of models.","marker":"[7]"},{"why":"Documents the missing model-selection criterion and random-projection sensitivity that motivate the method.","marker":"[1]"},{"why":"Supports the working assumption that ELMs are preferred on relatively small input-dimension problems.","marker":"[3]"},{"why":"Underlies the convergence of random neural networks to Gaussian processes, the basis of the NNGP kernel.","marker":"[9]"}],"fun_headline_variants":["Data-derived hidden weights match ELM accuracy","ELM without random weights: reproducible and faster","ENR-ELM replaces random hidden layer with eigenbasis","Optimal basis kills ELM randomness, keeps performance","New ELM uses eigenbasis, no random initialization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's load-bearing premise is that the kernel's eigenvectors can be well approximated by the hidden layer's fitted outputs, even though the construction guarantees no bound on the approximation error and the authors note that the exact oracle does not exist.","fun_headline_variants_meta":{"raw":{"variants":["Data-derived hidden weights match ELM accuracy","ELM without random weights: reproducible and faster","ENR-ELM replaces random hidden layer with eigenbasis","Optimal basis kills ELM randomness, keeps performance","New ELM uses eigenbasis, no random initialization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1555,"prompt_tokens":1029,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":451}},"tokens_in":645,"tokens_out":526,"duration_ms":5575,"temperature":1.0,"reasoning_tokens":451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:21:04.082788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on a small-input-dimension regression problem, the normalized Frobenius error $\\|\\sigma(\\hat{W}X)^\\top - U\\|_F/\\|U\\|_F$ along with the gap between A-ENR-ELM's test error and the oracle test error of regressing $y$ directly on $U$; if both gaps are large while the method still matches ELM's accuracy, the optimality claim for $\\hat{W}$ is not what carries the performance.","supporting_citations":[{"cited_title":"In: Proceedings of the 36th International Con- ference on Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the fast algorithm that evaluates the NNGP kernel matrix $K_\\sigma$ whose eigenbasis defines $U$."},{"cited_title":"Pattern Recognition 96, 106960 (2019) https: //doi.org/10.1016/j.patcog.2019.07.005","cited_arxiv_id":null,"evidence_quote":"Establishes the ELM-to-kernel correspondence that motivates using the NNGP kernel to fix the hidden layer."},{"cited_title":"Neural Comput and Applic 33, 15121–15144 (2021) https://doi.org/10.1007/s00521-021-06402-y","cited_arxiv_id":null,"evidence_quote":"Supports the working assumption that ELMs are preferred on relatively small input-dimension problems."}],"review_version":1}