REVIEW 4 major objections 5 minor 29 references
Recurrent Stochastic Configuration Networks with Hybrid Regularization for Nonlinear Dynamics Modelling
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Two-stage regularization — LASSO for delays, L2 for the residual learner — is claimed to give recurrent stochastic configuration networks the universal approximation property and the best test error on three nonlinear dynamics tasks.
desk verdict The hybrid LASSO+RSCN idea is a reasonable engineering combination, but the universal approximation theorem has a load-bearing gap: the global ridge solution is not guaranteed to have smaller unregularized error than the constructive update. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The hybrid model's machinery is a two-stage cascade: LASSO (11) selects the delay variables, and the residuals $\hat{Y}=T-Y_{\mathrm{LASSO}}$ are learned by an incrementally built reservoir whose new node must satisfy inequality (14) with the L2-shifted constants $(b_g^2+C)^2/(b_g^2+2C)$, and whose output weight is set by (15), so that Theorem 1's contraction bound $\|e_{N+1}\|^2 \le (r+\mu_{N+1})\|e_N\|^2$ delivers the universal approximation result. The echo state property is inherited from the lower-triangular reservoir matrix (5)-(6), giving the network a stable dynamic response to small input variations.
What would settle it
Construct a case with two training samples and one existing reservoir node whose current output weight already makes the residual exactly zero, then add one candidate node and compute both $\|\tilde{e}_{N+1}\|^2$ from the global ridge solution (17)-(18) and $\|e_{N+1}\|^2$ from the single-node update (15); the proof needs $\|\tilde{e}_{N+1}\|^2 \le \|e_{N+1}\|^2$, and a simple numerical example will show whether that inequality holds for any $C>0$.
Extended reading notes
Core claim
On its own terms, the central claim is that the hybrid LASSO-RSCN-L2 model — LASSO for delay-order selection, then a recurrent stochastic configuration network with an L2 penalty fit to the residual target — has the universal approximation property and achieves the best testing NRMSE among the compared models on all three datasets. Theorem 1 states that if each added node satisfies inequality (14) and its output weight is set by (15), then the residual norm obeys $\|e_{N+1}\|^2 \le (r+\mu_{N+1})\|e_N\|^2$ with $\mu_{N+1} \to 0$, so $\lim_{N \to \infty}\|e_N\| = 0$. The paper further claims that the L2 penalty plus a projection-based online update keeps the output weights bounded and lets the network adapt quickly to new samples, while the triangular reservoir matrix preserves the echo state property.
Load-bearing premise
The proof of Theorem 1 assumes that the global ridge-regression solution always has unregularized squared error no larger than the constructive one-node update; this can fail when the current network already fits the training data well, because the L2 penalty shrinks every weight and can increase the unregularized error.
Editorial extensions
If this is right
- If the theorem holds, the construction yields a reservoir whose training error provably goes to zero without backpropagation, as long as each newly added node passes the supervisory inequality (14).
- The LASSO pre-filter gives an interpretable selection of which lagged inputs matter, and on the reported tasks the chosen delays align with the governing dynamics, while the reservoir size shrinks compared to plain ESN and RSCN baselines.
- The L2 penalty combined with the projection-based update keeps the output weights regularized and adaptable, which the paper argues mitigates the ill-posedness of temporal data with unknown orders.
- The empirical claim, if correct, is that LASSO-RSCN-L2 posts the lowest testing NRMSE across all three tasks, with notably fewer reservoir nodes than the non-LASSO alternatives.
- The two-stage residual-compensation idea is presented as a general strategy: first capture the linear lag structure, then let the recurrent randomized learner model what the linear part misses.
Reading between the lines
- If the flagged proof gap is real, the universal-approximation theorem as stated may not be established, yet the empirical gains could still hold through the early-stopping and regularization mechanism rather than through the specific contraction inequality.
- The two-stage residual idea could be transplanted to other reservoir models, such as leaky-integrator echo state networks, to test whether the LASSO-plus-residual compensation, not the RSCN construction itself, is what drives the reported accuracy.
- A natural extension is to replace plain LASSO with group-sparse or adaptive LASSO so that groups of lagged inputs are selected together, which could improve stability on correlated industrial variables.
- A direct ablation — LASSO-RSCN-L2 versus a single-stage RSCN with both L1 and L2 penalties on the same tasks — would isolate how much of the gain comes from the LASSO pre-filter rather than from the L2 regularizer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hybrid recurrent stochastic configuration network (LASSO-RSCN-L2) for nonlinear dynamics modelling. It first uses LASSO to select lagged order variables, then incrementally builds an RSCN whose output weights are obtained by an L2-regularized least-squares solve, and finally applies an online projection update for the output weights. The central theoretical claim is a universal approximation theorem (Theorem 1) asserting that the regularized incremental construction drives the training residual to zero. The experimental section reports NRMSE comparisons on a nonlinear system identification benchmark and two industrial datasets (debutanizer soft sensing and short-term power load forecasting), claiming that the proposed model achieves the lowest testing NRMSE.
Significance. The hybrid order-selection plus residual-compensation idea is practically motivated, and the paper gives a complete algorithmic description with mean/variance statistics over 50 trials, which facilitates reimplementation. If the universal approximation guarantee were valid, the regularized supervisory node-selection mechanism would be a useful extension of stochastic configuration network theory. However, the proof of Theorem 1 contains a false inference that is load-bearing for the main theoretical claim, and the experimental comparisons use different input-feature sets for the LASSO variants than for the baselines. As a result, neither the theoretical nor the empirical headline claims are established as stated.
major comments (4)
- [Section III-B, proof of Theorem 1, text after Eq. (18)] The assertion "It is easily inferred that ||tilde e_{N+1}||^2 <= ||e_{N+1}||^2" is false. The global solution of (17)-(18) minimizes the regularized objective, not the unregularized squared error, so a minimizer of the regularized problem can have larger unregularized residual than the constructive update (15) analyzed in the proof. For example, with one training sample (x,t)=(1,1), C=0.1, previous ridge weight 1/1.1, and new node g=2, inequality (14) holds with r=0.9, mu=0, and b_g=2; the constructive update gives residual squared about 4.9e-6, while the global ridge solution (18) gives residual squared about 3.8e-4. Therefore inequality (19) does not follow, and Theorem 1 does not establish the claimed limit for Algorithm 1, which uses the global ridge solution at every step.
- [Section III-B, Eqs. (14)-(18) and Algorithm 1] There is a mismatch between the object analyzed in Theorem 1 and the object used in the proposed algorithm. The theorem proves a contraction property for the constructive single-node output weight (15), but Algorithm 1 (line 29 and Step 7 of the textual description) recomputes all output weights by the global regularized least-squares formula (17)-(18). The node-selection criterion (25) is also derived from the constructive update. Consequently, even if inequality (14) were sufficient for the constructive construction, it does not control the residual of the global solution, so the convergence proof does not apply to the algorithm as stated.
- [Section IV-A and IV-B, experimental setup] The baseline comparison is not matched in input features. In the nonlinear system identification task, ESN, RSCN, and their variants predict y(n+1) from [y(n), u(n)], while the LASSO-based frameworks are given ten-order delays for the inputs. In the debutanizer case, non-LASSO models use [u1(n),...,u5(n), y(n-1)], whereas the LASSO variants implement second-order delays for every input and then select variables such as u1(n-2) and u4(n-2) from that larger set. The reported improvements of LASSO-RSCN-L2 over RSCN-L2 may therefore be due to the additional lagged inputs rather than to the hybrid regularization or the RSCN architecture, so the empirical claim that the proposed method outperforms other models is not cleanly supported.
- [Contributions (item 3) and Remarks 1 and 3] The paper claims that the echo state property is 'naturally inherited' and that convergence of the projection updates is guaranteed, but for both statements the proof is delegated to the authors' unpublished preprint [22]. Since [22] is not part of the peer-reviewed record, these guarantees cannot be independently verified from the manuscript, and the reliance on an unpublished source for a stated contribution weakens the support for the stability and convergence claims.
minor comments (5)
- [Algorithm 1, line 11] The line 'Set mu_{N+1} = (1-r)/N + 1;' is ambiguous; with standard operator precedence it means ((1-r)/N)+1, which tends to 1 and would violate the assumption lim mu_{N+1}=0 and mu_{N+1} <= (1-r). If the intended value is (1-r)/(N+1), it should be written with brackets.
- [References] Reference [23] is a duplicate of reference [5]; they list the same paper by Kamalapurkar, Fischer, Obuz, and Dixon with identical title, journal, volume, pages, and year.
- [Eqs. (8) and (9)] The inner product notation <e_{N,q}, g_{N+1}> is used before it is formally introduced; the authors should state explicitly that these are Euclidean inner products over the training samples.
- [Eq. (27)] The summation in (27) contains garbled notation ('dK_B d' and 'pdk_B d'); the index ranges and the meaning of the coefficient should be written out clearly.
- [Table I and Figs. 4 and 11] Table I lists ranges for reservoir size, sparsity, and scaling factor, but not for the regularization coefficient C, even though the text states that grid search is used for network parameters; Figs. 4 and 11 also omit axis labels for the vertical axis, making the reported optimal combinations hard to read.
Circularity Check
No significant circularity: the universal-approximation proof is an attempted independent derivation and the empirical claims are benchmarked on held-out data; the self-citations to [22] are ancillary, and the main proof gap is a correctness issue rather than a circular reduction.
full rationale
The claimed universal approximation theorem (Section III-B, Eqs. 13–19) is an attempted first-principles derivation. The contraction inequality before Eq. (16) follows algebraically from the constructive weight (15) and the supervisory inequality (14), so the proof is not simply re-stating the conclusion. The residual-learning architecture (Section III-A, Eq. 12) is a two-stage fit on training data, and the performance claims in Tables II and IV are measured on held-out testing sets, so the empirical predictions are not fitted inputs renamed as predictions. The main self-citations are to the authors' own prior work [22] for the echo state property (Remark 1) and projection-algorithm convergence (Remark 3); these properties are ancillary to the universal approximation theorem and to the external benchmarks, so they do not make the central claim circular. I do flag two material caveats that are not circularity: (i) the step "It is easily inferred that ∥˜e_{N+1}∥^2 ≤ ∥e_{N+1}∥^2" after Eq. (18) is an unsupported monotonicity claim for the global ridge solution and is a correctness risk; and (ii) the paper itself admits in Section V that the framework lacks a theoretical basis for selecting the regularization coefficient C. Neither caveat is a self-definitional equivalence, a fitted parameter renamed as a prediction, or a load-bearing self-citation chain, so the circularity score remains low.
Assumptions & free parameters
free parameters (5)
- L2 regularization coefficient C =
0.001 (System ID), 0.01 (Case 1), 0.001 (Case 2)
- LASSO regularization coefficient C_L =
Not reported numerically
- Reservoir size N =
87 (System ID), 28 (Case 1), 36 (Case 2)
- Scaling factor alpha and reservoir sparsity =
Ranges in Table I
- Random parameter range lambda and contractive sequence r =
lambda in {0.1, 0.5, 1, 5, 10, 30, 50, 100}; r in [0.9, 0.99999]
assumptions (4)
- domain assumption span(Gamma) is dense in L2 space
- domain assumption The echo state property is inherited from [22]
- domain assumption Convergence of the projection algorithm is guaranteed as analyzed in [22]
- ad hoc to paper Random search finds a node satisfying inequality (14)
Cite this review
Pith. "Pith review of Recurrent Stochastic Configuration Networks with Hybrid Regularization for Nonlinear Dynamics Modelling." pith.science (2026). https://pith.science/paper/XW76W7XO
@misc{pith2026241200070,
author = {Pith},
title = {Pith review of: Recurrent Stochastic Configuration Networks with Hybrid Regularization for Nonlinear Dynamics Modelling},
year = {2026},
howpublished = {\url{https://pith.science/paper/XW76W7XO}},
note = {Machine review of arXiv:2412.00070}
}
read the original abstract
Recurrent stochastic configuration networks (RSCNs) have shown great potential in modelling nonlinear dynamic systems with uncertainties. This paper presents an RSCN with hybrid regularization to enhance both the learning capacity and generalization performance of the network. Given a set of temporal data, the well-known least absolute shrinkage and selection operator (LASSO) is employed to identify the significant order variables. Subsequently, an improved RSCN with L2 regularization is introduced to approximate the residuals between the output of the target plant and the LASSO model. The output weights are updated in real-time through a projection algorithm, facilitating a rapid response to dynamic changes within the system. A theoretical analysis of the universal approximation property is provided, contributing to the understanding of the network's effectiveness in representing various complex nonlinear functions. Experimental results from a nonlinear system identification problem and two industrial predictive tasks demonstrate that the proposed method outperforms other models across all testing datasets.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[22]
Recurrent stochastic configuration networks for temporal data analytics,
D. Wang and G. Dang, “Recurrent stochastic configuration networks for temporal data analytics,” arXiv: 2406.16959v1, Jun. 2024
arXiv 2024
-
[1]
A survey on deep learning for data-driven soft sensors,
Q. Sun and Z. Ge, “A survey on deep learning for data-driven soft sensors,” IEEE Trans. Ind. Inf., vol. 17, no. 9, pp. 5853-5866, Sept. 2021
work page 2021
-
[2]
Fuzzy adaptive knowledge-based inference neural networks: design and anal- ysis,
S. Liu, S.-K. Oh, W. Pedrycz, B. Yang, L. Wang, and K. Seo, “Fuzzy adaptive knowledge-based inference neural networks: design and anal- ysis,” IEEE Trans. Cybern., vol. 54, no. 9, pp. 4875-4888, Sept. 2024
work page 2024
-
[3]
A learning con- volutional neural network approach for network robustness prediction,
Y . Lou, R. Wu, J. Li, L. Wang, X. Li, and G. Chen, “A learning con- volutional neural network approach for network robustness prediction,” IEEE Trans. Cybern., vol. 53, no. 7, pp. 4531-4544, Jul. 2023
work page 2023
-
[4]
Identification of nonlinear output-affine systems using an orthogonal least squares algorithm,
S. Billings, M. Korenberg, and S. Chen, “Identification of nonlinear output-affine systems using an orthogonal least squares algorithm,” Int. J. Syst. Sci., vol. 19, no. 8, pp. 1559–1568, Apr. 1988
work page 1988
-
[6]
J. Lu, J. Ding, X. Dai and T. Chai, “Ensemble stochastic configuration networks for estimating prediction intervals: a simultaneous robust training algorithm and its application,” IEEE Trans. Neural Networks Learn. Syst., vol. 31, no. 12, pp. 5426-5440, Dec. 2020
work page 2020
-
[7]
Recurrent neural networks and long shortterm memory networks: Tutorial and survey,
B. Ghojogh and A. Ghodsi, “Recurrent neural networks and long shortterm memory networks: Tutorial and survey,” arXiv:2304.11461, 2023
arXiv 2023
-
[8]
X. Gao, L. Yan, G. Wang and C. Gerada, “Hybrid recurrent neural network architecture-based intention recognition for human–robot col- laboration,” IEEE Trans. Cybern., vol. 53, no. 3, pp. 1578-1586, Mar. 2023
work page 2023
Show all 29 references
-
[9]
Recurrent neural network training with convex loss and regularization functions by extended kalman filtering,
A. Bemporad, “Recurrent neural network training with convex loss and regularization functions by extended kalman filtering,” IEEE Trans. Autom. Control , vol. 68, no. 9, pp. 5661-5668, Sept. 2023
2023
-
[10]
Functional-link net computing: Theory, system architecture, and functionalities,
Y .-H. Pao and Y . Takefuji, “Functional-link net computing: Theory, system architecture, and functionalities,” Computer, vol. 25, no. 5, pp. 76–79, May 1992
1992
-
[11]
The echo state approach to analysing and training recurrent neural networks-with an erratum note,
H. Jaeger, “The echo state approach to analysing and training recurrent neural networks-with an erratum note,” German Nat. Res. Center Inf. Technol., Bonn, Germany, Tech. Rep. GMD, 148, 2001
2001
-
[12]
Randomness in neural networks: an overview,
S. Scardapane and D. Wang, “Randomness in neural networks: an overview,” Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery, vol. 7, no. e1200, Feb. 2017
2017
-
[13]
Editorial: Randomized algorithms for training neural net- works,
D. Wang, “Editorial: Randomized algorithms for training neural net- works,” Inf. Sci., vols. 364–365, pp. 126–128, Oct. 2016
2016
-
[14]
Optimization and applications of echo state networks with leaky-integrator neurons,
H. Jaeger, M. Lukosevicius, D. Popovici, and U. Siewert, “Optimization and applications of echo state networks with leaky-integrator neurons,” Neural Networks, vol. 20, no. 3, pp. 35-352, Apr. 2007
2007
-
[15]
Pruning and regularization in reservoir computing,
X. Dutoit, B. Schrauwen, J. Campenhout, D. Stroobandt, H. Van Brussel, and M. Nuttin, “Pruning and regularization in reservoir computing,” Neurocomputing, vol. 72, no. 7-9, pp. 1534-1546, Mar. 2009
2009
-
[16]
Dynamical regularized echo state network for time series prediction,
C. Yang, J. Qiao, L. Wang, and X. Zhu, “Dynamical regularized echo state network for time series prediction,” Neural Comput. Appl., vol. 31, pp. 6781-6794, May 2018
2018
-
[17]
A decentralized training algorithm for echo state networks in distributed big data applications,
S. Scardapane, D. Wang, and M. Panella, “A decentralized training algorithm for echo state networks in distributed big data applications,” Neural Networks, vol. 78, pp. 65-74, Jun. 2016
2016
-
[18]
Stochastic configuration networks: Fundamentals and algorithms,
D. Wang and M. Li, “Stochastic configuration networks: Fundamentals and algorithms,” IEEE Trans. Cybern., vol. 47, no. 10, pp. 3466-3479, Oct. 2017
2017
-
[19]
Multitarget stochastic config- uration network and applications,
Q. Wang, Q. Hong, S. Wu, and W. Dai, “Multitarget stochastic config- uration network and applications,” IEEE Trans. Artif. Intell., vol. 4, no. 2, pp. 338-348, Apr. 2023
2023
-
[20]
A sparse learning method for SCN soft measurement model,
K. Sun, L. Zhao, P. Tian, J. Zhao, D. Wang, “A sparse learning method for SCN soft measurement model,” Inf. Sci., vol. 659, 120098, Feb. 2024
2024
-
[21]
A regularized stochastic configuration network based on weighted mean of vectors for regres- sion,
Y . Wang, T. Zhou, G. Yang, C. Zhang, S. Li, “A regularized stochastic configuration network based on weighted mean of vectors for regres- sion,” PeerJ Comput. Sci., vol. 9, 1382, May 2023
2023
-
[23]
Time-varying input and state delay compensation for uncertain nonlinear systems,
R. Kamalapurkar, N. Fischer, S. Obuz and W. E. Dixon, “Time-varying input and state delay compensation for uncertain nonlinear systems,” IEEE Trans. Autom. Control, vol. 61, no. 3, pp. 834-839, Mar. 2016
2016
-
[24]
Dynamic gain reduced-order observer- based global adaptive neural-network tracking control for nonlinear time-delay systems,
W. Li, Z. Zhang and S. S. Ge, “Dynamic gain reduced-order observer- based global adaptive neural-network tracking control for nonlinear time-delay systems,” IEEE Trans. Cybern., vol. 53, no. 11, pp. 7105- 7114, Nov. 2023
2023
-
[25]
Regression shrinkage and selection via the lasso,
R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. R. Stat. Soc. Ser. B-Stat. Methodol., vol. 58, no. 1, pp. 267–288, 1996
1996
-
[26]
Regularization and variable selection via the elastic net,
H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,” J. R. Stat. Soc. Ser. B-Stat. Methodol., vol. 67, no. 2, pp. 301–320, 2005
2005
-
[27]
Predicting particle size of copper ore grinding with stochastic configuration networks,
D. Wang, P. Tian, W. Dai, and G. Yu, “Predicting particle size of copper ore grinding with stochastic configuration networks,” IEEE Trans. Ind. Inf., Early access, (online available), Jul. 2024
2024
-
[28]
Adaptive filtering prediction and control,
G. Goodwin and K. Sin, “Adaptive filtering prediction and control,” Courier Corporation, 2014
2014
-
[29]
Soft sensors for product quality monitoring in debutanizer distillation columns,
L. Fortuna, S. Graziani, and M. G. Xibilia, “Soft sensors for product quality monitoring in debutanizer distillation columns,” Control Eng. Pract., vol. 13, no. 4, pp. 499-508, Apr. 2005
2005
-
[30]
Recurrent stochastic configuration networks with incremental blocks,
G. Dang and D. Wang, “Recurrent stochastic configuration networks with incremental blocks,” arXiv: 2411.11303v1, Nov. 2024
2024 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.