{"id":"3eabb900-eb1d-449e-9d2d-71217d0db39b","arxiv_id":"1908.08450","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Echo State Network k-fold cross-validation can be implemented with O(Nr^2 T + k Nr^3) time, so the dominant training cost stays constant in k for long time series.","lead":"This paper shows how to run k-fold cross-validation for Echo State Networks at almost the same cost as a single train-validate split, by reusing one pass over the data. It also compares several time-aware validation schemes on real datasets and finds no single best scheme.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficient algorithm's subtraction identity is not a faithful holdout k-fold CV: post-validation training states in the single full pass are computed using validation-fold inputs (and teacher-forced targets), so the training gram differs from a clean rerun over the remaining folds.","rationale":"The paper's central complexity claim is mathematically sound as a statement about computing readouts on a fixed feature matrix: global-minus-fold gram subtraction is a valid linear algebra identity, and Eq. (9) follows for that procedure. The reader correctly identified the consecutive-fold and no-feedback conditions as necessary premises. However, a more load-bearing issue is the statistical validity of the procedure as k-fold CV. The reservoir is a stateful, recurrent feature extractor; states in a training block that follows a validation fold in time were computed using inputs from that validation fold. In generative tasks, where inputs are teacher-forced targets (Section 2.2), this means held-out labels influence the training features, producing optimistically biased validation estimates. This is not captured by the reader's weakest assumption, which focuses on the mechanics of fold arrangement rather than on the meaning of 'training on all other folds.' The paper does not disclose or measure this discrepancy. The empirical results in Table 2 are consistent with this leakage: k-fold CV performs worse than AV/FV, which could be due to the leakage-induced optimism rather than (or in addition to) nonstationarity. The concern does not overturn the paper's algorithmic contribution, since the method may still be useful as a feature-splitting CV or an approximation, and the complexity claim stands for that interpretation. But the paper should either reframe the method as an approximation or explicitly justify why the state leakage is negligible for the tested regimes. Hence the verdict remains CONDITIONAL, as the reader concluded, but the conditions should include addressing this leakage.","tokens_in":8858,"tokens_out":16392,"duration_ms":177536,"concrete_test":"On a generative dataset (e.g., Sunspots), compare the efficient subtraction CV with a clean holdout implementation: for each middle-fold split, rerun the reservoir over only the training folds, carrying the state across the skipped validation fold using the state at the end of the preceding training segment, and recompute the training gram and validation NRMSE. Repeat for a long-memory reservoir (rho=1.5, alpha=0.1) and a short-memory one (rho=0.1, alpha=1). If middle-fold validation errors differ systematically between the two implementations, the subtraction identity is not a faithful k-fold CV.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.5 proposes computing the training gram for each split as the global XX^T and YtargetX^T minus the validation-fold contributions, using reservoir states from a single full pass over the whole sequence. For a split whose validation fold lies in the middle, the training block after that fold uses states x(n) that were generated in the full pass using the validation-fold inputs. In generative tasks, Section 2.2 states that inputs are teacher-forced targets, so the held-out validation targets directly influence the states used to train the readout on later training blocks. The subtraction identity is exact only for the fixed full-run feature matrix, not for the k-fold CV defined in Section 2.3, where training should be performed on all other folds without seeing validation data. The paper neither acknowledges this discrepancy nor quantifies its effect; the complexity claim in Eq. (9) describes the subtraction procedure, but that procedure is a different (and potentially optimistically biased) cross-validation scheme. This is an internal consistency issue, not merely a departure from common practice: the defined CV scheme and the efficient implementation are not equivalent when the reservoir has finite memory and validation folds precede training blocks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes several temporal cross-validation schemes for echo state networks (CV, AV, FV, and their k-step variants) and an efficient implementation that avoids rerunning the reservoir for every split. The central idea is to accumulate the global gram matrices XX^T and YtargetX^T in one pass through the data and, for each split, subtract the validation fold's contribution before solving the ridge regression. This leads to the claimed complexity O(Nr^2 T + k Nr^3) in Eq. (9), i.e., no growth in the dominant reservoir pass with the number of folds k. The authors also report experiments on four generative time-series datasets and the Japanese Vowels classification benchmark, concluding that AV/FV schemes often give better test generalization than a single static split, and they discuss how the final model can be produced by retraining, averaging, or selection.","tokens_in":9051,"tokens_out":9180,"duration_ms":96479,"significance":"If the complexity claim held for the actual cross-validation procedure, this would be a practically valuable contribution: it would remove the main computational obstacle to routine k-fold validation of ESN hyperparameters. The paper's notation is clear, the complexity decomposition in Eqs. (7)-(9) is explicit, and the proposed schemes are sensible and well motivated. The empirical comparison on several real-world datasets, despite limitations, is a genuine attempt to evaluate the methods under different task types. No circularity is present: the complexity bound follows from the stated equations, and the paper does not reuse fitted constants in its derivation. However, as detailed below, the equivalence of the efficient algorithm to the defined CV scheme is not established for stateful time-series tasks, which is a load-bearing issue for the paper's central claim.","major_comments":[{"comment":"The subtraction identity in Section 2.5 is not equivalent to the k-fold CV defined in Section 2.3 when the validation fold lies before a training block. In the single full pass used to build XX^T and YtargetX^T, the reservoir state at a later training block is computed from the inputs (and, under the teacher forcing described in Section 2.2, the targets) of the earlier validation fold. Subtracting the validation fold's contribution from the global sum removes the direct contributions of those time steps, but it does not remove the influence of those time steps on the feature vectors of subsequent training points. The resulting regression is therefore trained on a different, temporally leaky design matrix. Eq. (9) states the complexity of this leaky procedure, not of the CV procedure of Section 2.3. The authors should either restrict the one-pass implementation to settings where this contamination is absent (e.g., independent sequences in the classification setting, or validation folds placed after all training data as in AV), or explicitly present the leaked procedure as an approximate CV scheme and quantify its bias.","section":"Section 2.5, Eq. (9)"},{"comment":"The empirical comparison for the generative datasets reports a single run per dataset, with no error bars or standard deviations. The claim that \"in all the experiments either FV or AV find the hyper-parameters producing best generalizing models\" is therefore not statistically supported, since random reservoir initialization alone can change the ranking. The paper also does not state the number of reservoir initializations, the washout length, or how fold-averaged validation errors are aggregated. Please report multiple seeds with mean and standard deviation, or at least clearly state that the results are illustrative rather than conclusive.","section":"Section 3, Table 2"},{"comment":"The claimed adaptation of the efficient algorithm to AV, FV, and k-step variants is not specified. For AV and FV, the training set of a split is a prefix or a fixed window, so the global-minus-validation-fold subtraction does not produce the required training gram; one would need fold-wise or rolling gram matrices. For the k-step variants, the overlapping validation blocks violate the \"consecutive folds\" assumption stated in Section 2.5. Since Table 2 and Figure 3 rely on k-step AV and FV, the paper should give the actual adapted algorithms and their complexity, or restrict the efficiency claim to the standard k-fold CV case.","section":"Section 2.5, Section 3, Figure 2"}],"minor_comments":[{"comment":"The phrase \"k-means cross-validation\" should read \"k-fold cross-validation\"; there is also a typo \"propsed\" in the same paragraph.","section":"Section 2.5"},{"comment":"The table title \"Japanese Wovels\" contains a typo and should be \"Japanese Vowels.\"","section":"Table 3"},{"comment":"The reference to \"Figure 2.1A\" is ambiguous; it should refer to a specific panel, e.g., \"Figure 2, panel 1A.\"","section":"Section 2.5"},{"comment":"The acronym NRMSE is used without a definition; please define it at first occurrence.","section":"Section 3.1"},{"comment":"The layout of Table 2 is hard to read: the grouping of the \"Averaged\", \"Retrained\", and \"Best\" rows under each scheme is not visually distinct, and the \"Validation Final Valid Test\" header is confusing. Consider using separate subtables or clearer horizontal rules.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on 1908.08450. The paper does two things: it lays out a taxonomy of validation schemes for ESNs (k-fold CV, accumulative, walk-forward, plus k-step variants), and it proposes an efficient implementation that reuses one full pass through the data to compute all fold-wise gram matrices. The complexity claim, O(Nr^2 T + k Nr^3), is attractive, and the gram-subtraction identity is exact for the fixed feature matrix.\n\nThe good stuff: the validation taxonomy is clearly presented and practically useful; the discussion of when CV helps in time series (scarce data, non-stationarity) is sensible; the experiments across several datasets are honest and show no single winning scheme, which is a useful negative result. The authors also discuss final model construction (retrain, average, best) with appropriate caveats.\n\nThe soft spot is real and load-bearing. The efficient algorithm runs the reservoir once through the whole sequence, then for each split subtracts the validation fold's contribution to the gram matrices. That is exact only if the feature matrix X is the same in the true CV as in the full pass. It isn't. For splits where a training block follows the validation fold, the full-pass reservoir states at those later times have been driven by the validation fold's inputs (and teacher-forced targets in generative tasks). A clean k-fold CV would train on those later blocks using states that never saw the validation data. So the efficient algorithm implements a slightly different CV scheme, one with potential optimistic bias, and the paper does not acknowledge or quantify the discrepancy. This affects the headline claim for CV. For the AV and FV schemes, where training is always before the validation fold, the full-pass states are clean, so the efficient implementation is faithful there.\n\nThe empirical section is also thin in places: Table 2 reports single runs with no error bars, and the adaptation to AV/FV is described in words without a concrete algorithm. These are more minor, since the main point is algorithmic.\n\nOverall, this is a useful practical paper with a fixable flaw. The authors should either provide an algorithm that avoids the contamination for CV (e.g., rerunning the reservoir for post-validation blocks, which would increase complexity for those splits) or explicitly reframe the efficient method as a new, slightly different validation scheme and compare its behavior to true CV empirically. I'd send it to peer review with that requirement.\n\nRecommendation: engage with it, but be ready to push on the faithfulness of the efficient CV.","headline":"Useful practical paper on fast cross-validation for ESNs, but the efficient algorithm silently changes the CV scheme by letting validation data leak into reservoir states for later training blocks.","tokens_in":9601,"tokens_out":4760,"would_cite":false,"duration_ms":43528,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-validating ESNs for roughly the cost of one split","keywords":["Echo State Networks","Reservoir Computing","Cross-Validation","Time Series","Time Complexity","Ridge Regression","Hyperparameter Tuning"],"falsifier":"Measure wall-clock time of the proposed k-fold algorithm on a long synthetic continuous series with $T$ much larger than $N_r$ (for example, $T = 10^6$, $N_r = 100$) for $k = 2, 10, 50$; if running time grows roughly linearly with k while k is well below $T/N_r$, the constant-in-k dominant term is not realized, and permuting the fold order so boundary states cannot be reused should make the speedup vanish.","tokens_in":8673,"feed_emoji":"⏱️","tokens_out":6862,"duration_ms":65555,"temperature":0.7,"pith_summary":"This paper claims that k-fold cross-validation of Echo State Networks can be made almost as cheap as a single train-validate pass. The standard approach reruns the reservoir k times, costing O(k $N_r^{2}$ T); the proposed method runs the reservoir once over the whole sequence, accumulates global Gram matrices, and subtracts each validation fold's contribution per split, giving O($N_r^{2}$ T + k $N_r^{3}$). When the time series is far longer than the network (T >> N_r) and k stays below T/N_r, the dominant cost does not grow with k and memory can remain O($N_r^{2}$). The authors introduce cross-validation, accumulative, and walk-forward schemes, and show empirically that they can select better hyperparameters than a single split, especially on non-stationary series.","feed_headline":"Cross-validate ESNs at roughly single-split cost","feed_subtitle":"A state-saving trick makes k-fold validation of reservoir networks no slower than one pass on long time series.","key_machinery":"The central objects are the global Gram matrices $XX^\\top$ and $Y_{\\mathrm{target}}X^\\top$, collected in one pass over the full sequence. Per split, the validation fold's contribution is subtracted from these global sums to form the ridge-regression training quantities, and the reservoir state saved at the fold boundary lets the next fold continue without a fresh run. This segment-decomposition identity, combined with the Markovian state update and the linear readout, is what carries the complexity saving and yields the bound $O(N_r^2T + kN_r^3)$.","core_discovery":"Cross-validating an ESN does not require rerunning the reservoir for every fold. Because the reservoir state is Markovian and the readout has no feedback into it, the state at each fold boundary can be saved during one full pass. The algorithm stores global $XX^\\top$ and $Y_{\\mathrm{target}}X^\\top$ once, then for each split either reuses or reruns only the validation fold, subtracts that fold's contribution to obtain the training Gram matrices, and solves ridge regression per split. This reduces the dominant time term from $O(kN_r^2T)$ to $O(N_r^2T)$, leaving only the per-split matrix inversions $O(kN_r^3)$. For $T \\gg N_r$ and $k < T/N_r$, k-fold cross-validation therefore has essentially the same time and space complexity as a single validation.","pith_inferences":["The same Gram-matrix subtraction trick should transfer to any linear-readout model whose hidden state evolves independently of the trained output, so long as validation folds remain contiguous blocks of one continuous series.","A direct stress test is to shuffle or randomize the folds: the predicted speedup should disappear because saved boundary states no longer connect folds, which would cleanly separate the algorithm's benefit from implementation noise.","The complexity bound suggests an adaptive strategy: for a single continuous series, use k up to roughly $T/N_r$ at no extra cost, effectively replacing single-split validation with full cross-validation in the data-rich regime.","In the complementary regime where k exceeds $T/N_r$, the $kN_r^3$ inversion term dominates, so the practical gain is confined to long-series settings; for short series the method still helps but the speedup is less pronounced."],"forward_implications":["For long time series with $T \\gg N_r$, the cost of k-fold cross-validation stays flat as k grows until k approaches $T/N_r$, so practitioners can use many folds for nearly the price of one validation.","The same time and space bound applies to any reservoir-type model with a linear readout and no output feedback, not only classical ESNs.","The proposed CV, AV, and FV schemes give concrete alternatives to static validation; the experiments suggest AV and FV select better forward-predicting models on drifting data, while CV improves hyperparameter estimates on scarce data.","The final trained model can be produced by retraining, averaging readout matrixes, or choosing the best-validated split, making the choice of final-model strategy a joint decision with the validation scheme."],"supporting_citations":[{"why":"Supplies the ridge-regression training equations, the on-the-fly Gram-matrix collection, and the $O(N_r^2T)$ single-pass complexity that the proposed algorithm exploits.","marker":"[6]"},{"why":"Establishes the echo state property, which justifies the short initialization and the carrying of reservoir states across validation-fold boundaries.","marker":"[7]"},{"why":"Provides the discussion of when cross-validation is valid for temporal time-series prediction, framing why the ESN case is simpler than general recurrent models.","marker":"[8]"},{"why":"Supplies the ESN classification model and benchmark result used to shape the empirical comparison on the Japanese Vowels task.","marker":"[10]"}],"fun_headline_variants":["ESN k-fold CV: one reservoir pass, all folds","State replay enables fast ESN cross-validation","Validate ESNs k-fold without rerunning the reservoir","Markovian state saves ESN validation time","k-fold ESN validation at single-split complexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The savings rely on the validation folds being consecutive blocks of one continuous time series, with the reservoir state saved at each boundary and carried into the next split; if folds are shuffled or non-contiguous, or the readout feeds back into the reservoir, the reservoir must be rerun and the $O(N_r^2T + kN_r^3)$ bound no longer follows.","fun_headline_variants_meta":{"raw":{"variants":["ESN k-fold CV: one reservoir pass, all folds","State replay enables fast ESN cross-validation","Validate ESNs k-fold without rerunning the reservoir","Markovian state saves ESN validation time","k-fold ESN validation at single-split complexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2506,"prompt_tokens":890,"completion_tokens":1616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1540}},"tokens_in":506,"tokens_out":1616,"duration_ms":13692,"temperature":1.0,"reasoning_tokens":1540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:40:15.016882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure wall-clock time of the proposed k-fold algorithm on a long synthetic continuous series with $T$ much larger than $N_r$ (for example, $T = 10^6$, $N_r = 100$) for $k = 2, 10, 50$; if running time grows roughly linearly with k while k is well below $T/N_r$, the constant-in-k dominant term is not realized, and permuting the fold order so boundary states cannot be reused should make the speedup vanish.","supporting_citations":[{"cited_title":"A practical guide to applying echo state networks","cited_arxiv_id":null,"evidence_quote":"Supplies the ridge-regression training equations, the on-the-fly Gram-matrix collection, and the $O(N_r^2T)$ single-pass complexity that the proposed algorithm exploits."},{"cited_title":"Yildiz, Herbert Jaeger, and Stefan J","cited_arxiv_id":null,"evidence_quote":"Establishes the echo state property, which justifies the short initialization and the carrying of reservoir states across validation-fold boundaries."},{"cited_title":"A note on the validity of cross-validation for evaluating autoregressive time series prediction.Computational Statistics & Data Analysis , 120:70–83, 2018","cited_arxiv_id":null,"evidence_quote":"Provides the discussion of when cross-validation is valid for temporal time-series prediction, framing why the ESN case is simpler than general recurrent models."},{"cited_title":"Optimiza- tion and applications of echo state networks with leaky-integrator neurons","cited_arxiv_id":null,"evidence_quote":"Supplies the ESN classification model and benchmark result used to shape the empirical comparison on the Japanese Vowels task."}],"review_version":1}