Efficient Cross-Validation of Echo State Networks

Arnas Uselis (1) ((1) Kaunas University of Technology); Mantas Luko\v{s}evi\v{c}ius (1)

REVIEW 3 major objections 5 minor 10 references

Efficient Cross-Validation of Echo State Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cross-validating ESNs for roughly the cost of one split

desk verdict Useful practical paper on fast cross-validation for ESNs, but the efficient algorithm silently changes the CV scheme by letting validation data leak into reservoir states for later training blocks. read the letter →

arxiv 1908.08450 v1 pith:ZCJMFK3X submitted 2019-08-22 cs.LG stat.ML

Mantas Lukoševičius (1) , Arnas Uselis (1) ((1) Kaunas University of Technology) This is my paper · ORCID

classification cs.LGstat.ML

keywords EchoStateNetworksReservoirComputingCross-ValidationTimeSeriesComplexityRidgeRegressionHyperparameterTuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that k-fold cross-validation of Echo State Networks can be made almost as cheap as a single train-validate pass. The standard approach reruns the reservoir k times, costing O(k $N_r^{2}$ T); the proposed method runs the reservoir once over the whole sequence, accumulates global Gram matrices, and subtracts each validation fold's contribution per split, giving O($N_r^{2}$ T + k $N_r^{3}$). When the time series is far longer than the network (T >> N_r) and k stays below T/N_r, the dominant cost does not grow with k and memory can remain O($N_r^{2}$). The authors introduce cross-validation, accumulative, and walk-forward schemes, and show empirically that they can select better hyperparameters than a single split, especially on non-stationary series.

What carries the argument

The central objects are the global Gram matrices $XX^\top$ and $Y_{\mathrm{target}}X^\top$, collected in one pass over the full sequence. Per split, the validation fold's contribution is subtracted from these global sums to form the ridge-regression training quantities, and the reservoir state saved at the fold boundary lets the next fold continue without a fresh run. This segment-decomposition identity, combined with the Markovian state update and the linear readout, is what carries the complexity saving and yields the bound $O(N_r^2T + kN_r^3)$.

What would settle it

Measure wall-clock time of the proposed k-fold algorithm on a long synthetic continuous series with $T$ much larger than $N_r$ (for example, $T = 10^6$, $N_r = 100$) for $k = 2, 10, 50$; if running time grows roughly linearly with k while k is well below $T/N_r$, the constant-in-k dominant term is not realized, and permuting the fold order so boundary states cannot be reused should make the speedup vanish.

Watch

Extended reading notes

Core claim

Cross-validating an ESN does not require rerunning the reservoir for every fold. Because the reservoir state is Markovian and the readout has no feedback into it, the state at each fold boundary can be saved during one full pass. The algorithm stores global $XX^\top$ and $Y_{\mathrm{target}}X^\top$ once, then for each split either reuses or reruns only the validation fold, subtracts that fold's contribution to obtain the training Gram matrices, and solves ridge regression per split. This reduces the dominant time term from $O(kN_r^2T)$ to $O(N_r^2T)$, leaving only the per-split matrix inversions $O(kN_r^3)$. For $T \gg N_r$ and $k < T/N_r$, k-fold cross-validation therefore has essentially the same time and space complexity as a single validation.

Load-bearing premise

The savings rely on the validation folds being consecutive blocks of one continuous time series, with the reservoir state saved at each boundary and carried into the next split; if folds are shuffled or non-contiguous, or the readout feeds back into the reservoir, the reservoir must be rerun and the $O(N_r^2T + kN_r^3)$ bound no longer follows.

Editorial extensions

If this is right

For long time series with $T \gg N_r$, the cost of k-fold cross-validation stays flat as k grows until k approaches $T/N_r$, so practitioners can use many folds for nearly the price of one validation.
The same time and space bound applies to any reservoir-type model with a linear readout and no output feedback, not only classical ESNs.
The proposed CV, AV, and FV schemes give concrete alternatives to static validation; the experiments suggest AV and FV select better forward-predicting models on drifting data, while CV improves hyperparameter estimates on scarce data.
The final trained model can be produced by retraining, averaging readout matrixes, or choosing the best-validated split, making the choice of final-model strategy a joint decision with the validation scheme.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same Gram-matrix subtraction trick should transfer to any linear-readout model whose hidden state evolves independently of the trained output, so long as validation folds remain contiguous blocks of one continuous series.
A direct stress test is to shuffle or randomize the folds: the predicted speedup should disappear because saved boundary states no longer connect folds, which would cleanly separate the algorithm's benefit from implementation noise.
The complexity bound suggests an adaptive strategy: for a single continuous series, use k up to roughly $T/N_r$ at no extra cost, effectively replacing single-split validation with full cross-validation in the data-rich regime.
In the complementary regime where k exceeds $T/N_r$, the $kN_r^3$ inversion term dominates, so the practical gain is confined to long-series settings; for short series the method still helps but the speedup is less pronounced.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's my read on 1908.08450. The paper does two things: it lays out a taxonomy of validation schemes for ESNs (k-fold CV, accumulative, walk-forward, plus k-step variants), and it proposes an efficient implementation that reuses one full pass through the data to compute all fold-wise gram matrices. The complexity claim, O(Nr^2 T + k Nr^3), is attractive, and the gram-subtraction identity is exact for the fixed feature matrix.

The good stuff: the validation taxonomy is clearly presented and practically useful; the discussion of when CV helps in time series (scarce data, non-stationarity) is sensible; the experiments across several datasets are honest and show no single winning scheme, which is a useful negative result. The authors also discuss final model construction (retrain, average, best) with appropriate caveats.

The soft spot is real and load-bearing. The efficient algorithm runs the reservoir once through the whole sequence, then for each split subtracts the validation fold's contribution to the gram matrices. That is exact only if the feature matrix X is the same in the true CV as in the full pass. It isn't. For splits where a training block follows the validation fold, the full-pass reservoir states at those later times have been driven by the validation fold's inputs (and teacher-forced targets in generative tasks). A clean k-fold CV would train on those later blocks using states that never saw the validation data. So the efficient algorithm implements a slightly different CV scheme, one with potential optimistic bias, and the paper does not acknowledge or quantify the discrepancy. This affects the headline claim for CV. For the AV and FV schemes, where training is always before the validation fold, the full-pass states are clean, so the efficient implementation is faithful there.

The empirical section is also thin in places: Table 2 reports single runs with no error bars, and the adaptation to AV/FV is described in words without a concrete algorithm. These are more minor, since the main point is algorithmic.

Overall, this is a useful practical paper with a fixable flaw. The authors should either provide an algorithm that avoids the contamination for CV (e.g., rerunning the reservoir for post-validation blocks, which would increase complexity for those splits) or explicitly reframe the efficient method as a new, slightly different validation scheme and compare its behavior to true CV empirically. I'd send it to peer review with that requirement.

Recommendation: engage with it, but be ready to push on the faithfulness of the efficient CV.

Referee Report

3 major / 5 minor

Summary. The paper proposes several temporal cross-validation schemes for echo state networks (CV, AV, FV, and their k-step variants) and an efficient implementation that avoids rerunning the reservoir for every split. The central idea is to accumulate the global gram matrices XX^T and YtargetX^T in one pass through the data and, for each split, subtract the validation fold's contribution before solving the ridge regression. This leads to the claimed complexity O(Nr^2 T + k Nr^3) in Eq. (9), i.e., no growth in the dominant reservoir pass with the number of folds k. The authors also report experiments on four generative time-series datasets and the Japanese Vowels classification benchmark, concluding that AV/FV schemes often give better test generalization than a single static split, and they discuss how the final model can be produced by retraining, averaging, or selection.

Significance. If the complexity claim held for the actual cross-validation procedure, this would be a practically valuable contribution: it would remove the main computational obstacle to routine k-fold validation of ESN hyperparameters. The paper's notation is clear, the complexity decomposition in Eqs. (7)-(9) is explicit, and the proposed schemes are sensible and well motivated. The empirical comparison on several real-world datasets, despite limitations, is a genuine attempt to evaluate the methods under different task types. No circularity is present: the complexity bound follows from the stated equations, and the paper does not reuse fitted constants in its derivation. However, as detailed below, the equivalence of the efficient algorithm to the defined CV scheme is not established for stateful time-series tasks, which is a load-bearing issue for the paper's central claim.

major comments (3)

[Section 2.5, Eq. (9)] The subtraction identity in Section 2.5 is not equivalent to the k-fold CV defined in Section 2.3 when the validation fold lies before a training block. In the single full pass used to build XX^T and YtargetX^T, the reservoir state at a later training block is computed from the inputs (and, under the teacher forcing described in Section 2.2, the targets) of the earlier validation fold. Subtracting the validation fold's contribution from the global sum removes the direct contributions of those time steps, but it does not remove the influence of those time steps on the feature vectors of subsequent training points. The resulting regression is therefore trained on a different, temporally leaky design matrix. Eq. (9) states the complexity of this leaky procedure, not of the CV procedure of Section 2.3. The authors should either restrict the one-pass implementation to settings where this contamination is absent (e.g., independent sequences in the classification setting, or validation folds placed after all training data as in AV), or explicitly present the leaked procedure as an approximate CV scheme and quantify its bias.
[Section 3, Table 2] The empirical comparison for the generative datasets reports a single run per dataset, with no error bars or standard deviations. The claim that "in all the experiments either FV or AV find the hyper-parameters producing best generalizing models" is therefore not statistically supported, since random reservoir initialization alone can change the ranking. The paper also does not state the number of reservoir initializations, the washout length, or how fold-averaged validation errors are aggregated. Please report multiple seeds with mean and standard deviation, or at least clearly state that the results are illustrative rather than conclusive.
[Section 2.5, Section 3, Figure 2] The claimed adaptation of the efficient algorithm to AV, FV, and k-step variants is not specified. For AV and FV, the training set of a split is a prefix or a fixed window, so the global-minus-validation-fold subtraction does not produce the required training gram; one would need fold-wise or rolling gram matrices. For the k-step variants, the overlapping validation blocks violate the "consecutive folds" assumption stated in Section 2.5. Since Table 2 and Figure 3 rely on k-step AV and FV, the paper should give the actual adapted algorithms and their complexity, or restrict the efficiency claim to the standard k-fold CV case.

minor comments (5)

[Section 2.5] The phrase "k-means cross-validation" should read "k-fold cross-validation"; there is also a typo "propsed" in the same paragraph.
[Table 3] The table title "Japanese Wovels" contains a typo and should be "Japanese Vowels."
[Section 2.5] The reference to "Figure 2.1A" is ambiguous; it should refer to a specific panel, e.g., "Figure 2, panel 1A."
[Section 3.1] The acronym NRMSE is used without a definition; please define it at first occurrence.
[Table 2] The layout of Table 2 is hard to read: the grouping of the "Averaged", "Retrained", and "Best" rows under each scheme is not visually distinct, and the "Validation Final Valid Test" header is confusing. Consider using separate subtables or clearer horizontal rules.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the complexity result follows from the paper's stated linear algebra and is not obtained by reusing fitted constants or by a self-citation chain.

full rationale

The central claim, that k-fold cross-validation of ESNs can be performed in O(Nr^2 T + k Nr^3) time, is derived directly from the paper's own equations: the reservoir update (1)-(2), the ridge regression solution (6), the collection of gram matrices XX^T and YtargetX^T, and the algebraic subtraction of the validation-fold contributions from the global matrices in Section 2.5. This is an operation-count argument over explicitly stated matrix products and inversions, not a prediction that reduces to a fitted input or to a self-citation. The paper's citations to the authors' earlier ESN guides (e.g., [6]) supply background training details and notation, but the efficiency bound and the subtraction identity do not depend on any unverified claim imported from those references. A separate question is whether the subtractive procedure exactly reproduces the k-fold CV protocol as defined in Section 2.3, particularly for generative tasks where validation-fold inputs influence later reservoir states; that is a correctness or consistency concern, not a circularity concern, because the complexity result is about the algorithm the paper actually specifies. No step in the derivation chain is equivalent by construction to its own input, and no fitted parameter is renamed as a prediction. Thus the appropriate finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are introduced by the central complexity claim; the experimental hyperparameter grids are disclosed. The algorithm relies on standard ridge regression algebra plus the Markovian/echo-state property that makes a single reservoir pass sufficient. No invented entities are introduced.

assumptions (4)

standard math The ridge regression estimate (6) is the training objective and the gram matrices decompose additively over time columns: S_train = S_all - S_val.
Equation (6) and the subtraction identity in Section 2.5; this follows from the normal equations for ridge regression.
domain assumption Reservoir state x(n) is generated independently of the readout, so one forward pass over the whole sequence suffices to build all design matrices.
Assumed implicitly in Section 2.5; holds for the classical ESN (1)-(3) without output feedback.
domain assumption The echo state / Markovian property allows phase switches and fold-boundary states to be reused without reinitializing the reservoir.
Section 2.2: 'Because the memory of ESN is preserved in its collected state ... instant switches between phases'; needed for state carry-over between consecutive validation folds.
standard math Dense matrix inversion costs O(Nr^3) in practical implementations.
Used in deriving Eq. (9) for the per-fold ridge solves; standard complexity of Cholesky or similar linear algebra.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient Cross-Validation of Echo State Networks." pith.science (2026). https://pith.science/paper/ZCJMFK3X

@misc{pith2026190808450,
  author       = {Pith},
  title        = {Pith review of: Efficient Cross-Validation of Echo State Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCJMFK3X}},
  note         = {Machine review of arXiv:1908.08450}
}

abstract

Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cross-validating ESNs and introduce an efficient algorithm for implementing them. The component that dominates the time complexity of the already quite fast ESN training remains constant (does not scale up with $k$) in our proposed method of doing $k$-fold cross-validation. The component that does scale linearly with $k$ starts dominating only in some not very common situations. Thus in many situations $k$-fold cross-validation of ESNs can be done for virtually the same time complexity as a simple single split validation. Space complexity can also remain the same. We also discuss when the proposed validation schemes for ESNs could be beneficial and empirically investigate them on several different real-world datasets.

Figures

Figures reproduced from arXiv: 1908.08450 by the authors.

**Figure 1.** Splitting the data: a) all the available data; b) splitting-off the testing set; c) a static classical (SV) initialization, training, and validation split for ESNs; d) splitting data into folds 2 and up for n-fold cross-validation; e) the maximum amount of folds for leave-one-out cross-validation. To investigate the k-fold cross-validation of ESNs we split the data into kfolds. The number of folds k can be varied f… view at source ↗

**Figure 2.** Different validation schemes used. as is common in non-temporal tasks. CV is geared towards training the model once on a fixed well-representative dataset. AV emulates the classical static training and validation SV k times by each time training on all the available data that come before the validation fold and then validating. This scenario would also happen when the model is repeatedly updated with newly available… view at source ↗

**Figure 3.** Results of grid search on Electricity dataset. Every point corresponds to one combination of hyper-parameters [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages

[1]

echo state

Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networks. Technical Report GMD Report 148, German National Research Center for Information Technology, 2001

work page 2001
[2]

Harnessing nonlinearity: predicting chaotic sys- tems and saving energy in wireless communication.Science, 304(5667):78–80, 2004

Herbert Jaeger and Harald Haas. Harnessing nonlinearity: predicting chaotic sys- tems and saving energy in wireless communication.Science, 304(5667):78–80, 2004

work page 2004
[3]

Echo state network

Herbert Jaeger. Echo state network. Scholarpedia, 2(9):2330, 2007

work page 2007
[4]

Reservoir computing approaches to re- current neural network training

Mantas Lukoˇ seviˇ cius and Herbert Jaeger. Reservoir computing approaches to re- current neural network training. Computer Science Review , 3(3):127–149, August 2009

work page 2009
[5]

Cross-validatory choice and assessment of statistical predictions

Mervyn Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological) , 36(2):111–133, 1974

work page 1974
[6]

A practical guide to applying echo state networks

Mantas Lukoˇ seviˇ cius. A practical guide to applying echo state networks. In Gr´ egoire Montavon, Genevi` eve B. Orr, and Klaus-Robert M¨ uller, editors,Neural Networks: Tricks of the Trade, 2nd Edition , volume 7700 of LNCS, pages 659–686. Springer, 2012

work page 2012
[7]

Yildiz, Herbert Jaeger, and Stefan J

Izzet B. Yildiz, Herbert Jaeger, and Stefan J. Kiebel. Re-visiting the echo state property. Neural Networks, 35:1 – 9, 2012

work page 2012
[8]

A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis , 120:70–83, 2018

Christoph Bergmeir, Rob J Hyndman, and Bonsoo Koo. A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis , 120:70–83, 2018

work page 2018

Show all 10 references

[9]

Short-term electricity demand forecasting using double seasonal ex- ponential smoothing

J W Taylor. Short-term electricity demand forecasting using double seasonal ex- ponential smoothing. Journal of the Operational Research Society , 54(8):799–805, aug 2003

2003
[10]

Optimiza- tion and applications of echo state networks with leaky-integrator neurons

Herbert Jaeger, Mantas Lukoˇ seviˇ cius, Dan Popovici, and Udo Siewert. Optimiza- tion and applications of echo state networks with leaky-integrator neurons. Neural Networks, 20(3):335–352, 2007

2007

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

echo state

Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networks. Technical Report GMD Report 148, German National Research Center for Information Technology, 2001

work page 2001

[2] [2]

Harnessing nonlinearity: predicting chaotic sys- tems and saving energy in wireless communication.Science, 304(5667):78–80, 2004

Herbert Jaeger and Harald Haas. Harnessing nonlinearity: predicting chaotic sys- tems and saving energy in wireless communication.Science, 304(5667):78–80, 2004

work page 2004

[3] [3]

Echo state network

Herbert Jaeger. Echo state network. Scholarpedia, 2(9):2330, 2007

work page 2007

[4] [4]

Reservoir computing approaches to re- current neural network training

Mantas Lukoˇ seviˇ cius and Herbert Jaeger. Reservoir computing approaches to re- current neural network training. Computer Science Review , 3(3):127–149, August 2009

work page 2009

[5] [5]

Cross-validatory choice and assessment of statistical predictions

Mervyn Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological) , 36(2):111–133, 1974

work page 1974

[6] [6]

A practical guide to applying echo state networks

Mantas Lukoˇ seviˇ cius. A practical guide to applying echo state networks. In Gr´ egoire Montavon, Genevi` eve B. Orr, and Klaus-Robert M¨ uller, editors,Neural Networks: Tricks of the Trade, 2nd Edition , volume 7700 of LNCS, pages 659–686. Springer, 2012

work page 2012

[7] [7]

Yildiz, Herbert Jaeger, and Stefan J

Izzet B. Yildiz, Herbert Jaeger, and Stefan J. Kiebel. Re-visiting the echo state property. Neural Networks, 35:1 – 9, 2012

work page 2012

[8] [8]

A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis , 120:70–83, 2018

Christoph Bergmeir, Rob J Hyndman, and Bonsoo Koo. A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis , 120:70–83, 2018

work page 2018