{"id":"104a6221-de02-4008-97f5-13d1e800ddb8","arxiv_id":"2505.05916","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Feeding past prediction errors back into RNN, GRU, and LSTM hidden-state updates improves short-horizon forecasting accuracy on ETT benchmarks with only a 0.73% parameter increase.","lead":"This paper adds a correction signal to recurrent neural networks: the error from the previous prediction is fed back into the network at the next step, in the same spirit as a Kalman filter. On four electricity transformer datasets, the modified RNN, GRU, and LSTM variants forecast oil temperature with lower mean squared error than their standard versions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"At one-step prediction, ILSTM/IGRU/IRNN are reparameterizations of vanilla LSTM/GRU/RNN; the largest reported gains may be optimization artifacts, not innovation-driven improvements.","rationale":"The reader's weakest assumption was the zero-future-innovation approximation in Eq. (9). That is a real limitation of multi-step forecasting, but the more load-bearing issue is the unexamined algebraic equivalence at one step, where the paper reports its largest improvements. Because e_{t-1} is an affine function of the already-available variables x_{t-1} and y_{t-1}, every innovation term can be absorbed into the existing recurrent weights and biases. Consequently, IRNN/IGRU/ILSTM and their vanilla counterparts represent exactly the same set of 1-step input-output maps. The reported 1-step gains therefore do not demonstrate that 'the integration of innovations' improves accuracy; they may instead reflect differences in the optimization trajectory caused by IU-BPTT or by the particular parameterization. The multi-step regime (Tf > 0) is not equivalent because future innovations are set to zero, so the architecture has a plausible role there; however, the central claim is partly built on the 1-step results, and those need a control experiment. This does not show the method is useless, but it changes the required justification from 'innovation feedback improves modeling' to 'the proposed training procedure or parameterization helps optimization.' I therefore keep the verdict at CONDITIONAL, with the added condition that the equivalence control and variance reporting be supplied.","tokens_in":16946,"tokens_out":12908,"duration_ms":137391,"concrete_test":"Train the un-innovated vanilla LSTM obtained by substituting W_fx' = W_fx - W_fe W_yx, W_fy' = W_fy + W_fe, b_f' = b_f - W_fe b_y into every gate/cell of Table I, and similarly for IRNN/IGRU, using the same data splits, seeds, hyperparameters, and initialization (transformed from the ILSTM initialization) as in Section IV-A2; compare 1-step MSE over 20 runs with a paired test. If this equivalent model closes the reported 40-60% gap, the innovation input is not causally responsible for the headline gains.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section III defines e_{t-1} = y_{t-1} - ŷ_{t-1}, with ŷ_{t-1} = W_yx x_{t-1} + b_y affine in the hidden state. Substituting into Eq. (7), the IRNN update becomes x_t = σ((W_xx - W_xe W_yx)x_{t-1} + W_xu u_t + (W_xy + W_xe)y_{t-1} + (b_x - W_xe b_y)), which is exactly the vanilla RNN update (2) with reparameterized weights. The same absorption applies to every ILSTM gate and cell in Table I: replace each (W_fx, W_fy, b_f) by (W_fx - W_fe W_yx, W_fy + W_fe, b_f - W_fe b_y), and analogously for the input gate, output gate, and cell candidate. Thus, for 1-step prediction, the function class of IRNN/IGRU/ILSTM is identical to that of RNN/GRU/LSTM; the innovation term adds no representational capacity. The paper's most striking evidence is the 1-step column of Table IV, with 40-60% MSE reductions, yet these gains cannot be attributed to a larger function class. They must arise from IU-BPTT's alternating optimization (Eq. 16 freezes innovations when computing gradients) or from the parameterization/initialization, not from the KF-inspired feedback mechanism itself. The multi-step zero-innovation regime of Eq. (9) is acknowledged, but the one-step equivalence is not, and it undermines the causal reading of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an 'innovation-driven' recurrent neural network, where past prediction errors e_{t-1}=y_{t-1}-ŷ_{t-1} are fed back as additional inputs into the hidden-state updates of RNN, GRU, and LSTM, motivated by Kalman-filter innovation feedback. To train these networks, the authors introduce IU-BPTT, which alternates between standard BPTT parameter updates with the innovation trajectory frozen and recomputing innovations from the updated parameters. Experiments on the four ETT datasets report large one-step MSE reductions (40-60%) and smaller multi-step reductions (around 10% at five steps) over vanilla RNN/GRU/LSTM, with only a 0.73% parameter increase, together with an ablation study of which gates and cell modules benefit from innovations. The paper is clearly written and the training algorithm is precisely specified.","tokens_in":17366,"tokens_out":8985,"duration_ms":92665,"significance":"If the stated causal mechanism were established, the work would offer a simple and broadly applicable modification for recurrent time-series models. The IU-BPTT algorithm is practical and well described, the ablation study is thoughtful, and the ETT benchmark is standard. However, the central interpretation is weakened by an exact reparameterization equivalence between IRNN and the vanilla RNN at one-step horizons, and the empirical claims lack statistical support. The contribution remains potentially useful as an alternative training and data-augmentation scheme, but the paper must be reframed and supplemented with control experiments and significance testing before its claims are credible.","major_comments":[{"comment":"For one-step prediction, the innovation term is exactly absorbable into the vanilla RNN update. Substituting e_{t-1}=y_{t-1}-(W_yx x_{t-1}+b_y) into (7) gives x_t = σ((W_xx - W_xe W_yx)x_{t-1} + W_xu u_t + (W_xy + W_xe)y_{t-1} + (b_x - W_xe b_y)), which is the vanilla RNN recurrence (2) with reparameterized weights. The same substitution applies gate-by-gate to IGRU and ILSTM in Table I. Consequently, the 40-60% one-step MSE reductions in Table IV cannot be attributed to a larger function class or to innovation information; they must arise from the altered parameterization or from the IU-BPTT optimization trajectory. The authors should acknowledge this equivalence explicitly and add control experiments that separate architectural capacity from optimization effects, for example by training a vanilla LSTM under a corresponding reparameterized loss or by evaluating whether an equivalent vanilla network with different initialization achieves similar one-step accuracy.","section":"Section III-B, Eq. (7)"},{"comment":"The text states that all networks are trained 20 times, yet Table IV reports only point estimates of test MSE with no standard deviations, confidence intervals, or significance tests. Given the typical run-to-run variance of recurrent networks and the fact that several reported gains (especially at five steps) are only around 10%, these differences may not be statistically significant. The authors should report mean and standard deviation (or confidence intervals) over the 20 runs for each cell of Table IV and apply paired significance tests for the reported RNN-versus-IRNN comparisons.","section":"Table IV and Section IV-A2"},{"comment":"Equation (9) and Algorithm 1 (lines 9-11) set future innovations to zero in both training and evaluation, so the reported multi-step gains cannot come from online error correction during the prediction horizon; they must come from the way the initial context is processed using past innovations. The narrative in Section IV-B2 and Figure 4 that describes 'error-correcting capability' during multi-step prediction is therefore misleading. The authors should clarify that, in the multi-step setting, innovations act as a form of context preprocessing rather than closed-loop feedback, and the visualization should be labeled accordingly.","section":"Section III-B, Eq. (9) and Section IV-B2"}],"minor_comments":[{"comment":"\"attentions\" should be \"attention\" in the phrase \"have attracted immense attentions.\"","section":"Introduction, first paragraph"},{"comment":"The caption contains the typo \"Nertwork\"; it should read \"Network.\"","section":"Table VII caption"},{"comment":"The caption \"Each The legend indicates...\" should be corrected to \"The legend indicates...\"","section":"Figure 8 caption"},{"comment":"The name \"Naïve\" is written with a combining diaeresis in several places; use a consistent plain-text spelling such as \"Naive\" for uniformity.","section":"Throughout"},{"comment":"The input line \"Training setDT\" is missing a space; it should read \"Training set DT.\"","section":"Algorithm 1"},{"comment":"The abstract's claim that improvements come \"without increasing the training cost substantially\" is contradicted by Table VI for the default N=1, where per-epoch time increases by 15-20%; the claim should be qualified to the case of a large innovation-updating interval, as supported by Table VII.","section":"Abstract and Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"This paper is borderline for TKDE. The reparameterization equivalence at one-step horizons is the central concern: if the authors cannot demonstrate that the reported gains are causally tied to the innovation feedback rather than to the altered optimization, the contribution reduces to a training-scheme variant. The manuscript would benefit from the suggested control experiments and significance testing. The use of the authors' own prior work [24] as motivation is acceptable, but the novelty claim in Section III-A should be checked against the broader literature on error-feedback recurrent networks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the IRNN paper. The stress-test note is correct and it matters. For one-step prediction, feeding e_{t-1} is exactly equivalent to reparameterizing the vanilla RNN, GRU, or LSTM, because e_{t-1} is affine in the hidden state. So the 40-60% one-step improvements in Table IV cannot be attributed to a richer function class. They have to come from the alternating IU-BPTT training or from initialization. That undercuts the paper's causal story that the Kalman-inspired error feedback improves prediction.\n\nThat said, the paper has real substance. The IU-BPTT algorithm is clearly described and the ablation studies are unusually thorough. The authors correctly note that innovations are only computable when ground-truth outputs are available, so they restrict the method to time-series prediction. They also explicitly set future innovations to zero in multi-step prediction and show the gains decay with horizon, which is honest.\n\nThe soft spots, in order: the reparameterization equivalence is not acknowledged anywhere, so the central claim as stated is misleading. There are no standard deviations or significance tests despite 20 runs. The benchmark is only the four ETT datasets, so generalization is uncertain. And the paper never compares against classical error-feedback models like NARMAX or output-error neural networks, which are the natural baselines for this idea. On a smaller note, the training-time comparison in Table VI is somewhat apples-to-oranges because IU-BPTT includes the innovation-updating cost; the 10-15% overhead claim is not a clean measure of the algorithm's cost.\n\nMy take: the paper deserves a serious referee, because the optimization effect is interesting in its own right and the multi-step results are not explained by the reparameterization. But the authors need to reframe the contribution as a new training scheme for recurrent models, show variance bars, and discuss the equivalence (or demonstrate a setting where the function class provably differs, e.g., with nonlinear output layers or stacked architectures). If they can show that IU-BPTT consistently finds better optima, that is a publishable result even without a new architecture.\n\nWould I bring it to reading group? Maybe, as a case study in how easy it is to mistake an optimization artifact for a modeling innovation. I would not cite it yet.","headline":"The stress-test note is right: for one-step prediction IRNN/IGRU/ILSTM are reparameterizations of the vanilla models, so the large reported gains are likely optimization effects from IU-BPTT, not evidence for a richer function class; still, the paper is worth refereeing because the training scheme and multi-step behavior are non-trivial.","tokens_in":17853,"tokens_out":2840,"would_cite":false,"duration_ms":29001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62M20"],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving RNNs access to their own past prediction errors, the way Kalman filters use innovations, improves multi-step time-series forecasting accuracy on standard benchmarks while adding only 0.73 percent more parameters.","keywords":["recurrent neural networks","time series forecasting","Kalman filter","innovation","backpropagation through time","LSTM","GRU","multi-step prediction"],"falsifier":"A direct test is to train ILSTM under IU-BPTT with the innovation input $e_{t-1}$ replaced by a time-shifted or randomly permuted version of the same prediction error. If test MSE stays as low as with the true $e_{t-1}$, the gain is not the specific Kalman-style feedback; if MSE rises back toward LSTM levels, the immediate one-step error signal is the active component.","tokens_in":16728,"feed_emoji":"📈","tokens_out":9207,"duration_ms":81236,"temperature":0.7,"pith_summary":"This paper proposes that a recurrent neural network should be told about its own recent prediction mistakes, not just about the raw input history. Borrowing the Kalman filter's notion of innovation, the one-step-ahead error between the true output and the predicted output, the authors feed that error back into the hidden-state update, and into the gates and memory cells of LSTM and GRU. Because these innovation inputs depend on the network parameters, ordinary backpropagation through time cannot train the model directly, so the paper introduces input-updating BPTT (IU-BPTT), which alternates between refreshing the innovation signals and taking gradient steps. On four electricity transformer temperature datasets, the innovation-driven variants consistently beat their plain counterparts, with ILSTM reducing test MSE by roughly 19 to 31 percent while increasing parameter count by less than one percent. If this holds, it means a small architectural change, error feedback rather than a larger model, is enough to lift recurrent forecasting accuracy.","feed_headline":"RNNs that feed back prediction errors cut forecast error by 31%","feed_subtitle":"Innovation feedback gives LSTM 19-31 percent lower test MSE on four ETT datasets.","key_machinery":"The load-bearing object is the innovation signal $e_t = y_t - \\hat y_t$, the immediate prediction error, fed back into the recurrence through a learnable gain matrix, together with the IU-BPTT training loop that alternates between refreshing innovations and updating parameters. In the LSTM and GRU versions, the same error enters the forget, input, output, reset, and update gates as well as the cell-state or candidate-state computation. For multi-step prediction the paper sets future innovations to zero, so the recursive predictor runs exactly like the base RNN beyond the first step; the mechanism is therefore most active in one-step-ahead and teacher-forced regimes.","core_discovery":"The central claim is that injecting past prediction errors $e_{t-1}=y_{t-1}-\\hat y_{t-1}$ as additional inputs into RNN, GRU, and LSTM recurrences turns them into self-correcting predictors, analogous to the Kalman filter's use of innovations in linear state-space models. In single-layer IRNN the learnable matrix $W_{xe}$ multiplying $e_{t-1}$ plays the Kalman-gain role. The same innovation signal is added to the gates and cell candidate of GRU and LSTM, yielding IGRU and ILSTM. Because innovations are functions of the current parameters, the paper trains with IU-BPTT, alternating parameter-gradient updates with recomputation of innovations. Experiments on ETTh1, ETTh2, ETTm1, and ETTm2 show that the innovation-driven variants improve multi-step MSE across the board, that ILSTM gives the largest gains with roughly 31 percent average reduction on ETTh1 and ETTh2, and that the architecture only adds 0.73 percent parameters.","pith_inferences":["A natural extension is to replace the zero-future-innovation assumption with a learned or autoregressive model of future innovations; the paper's own decay of gains from one-step to five-step horizons suggests such a model would be the main lever for longer-horizon gains.","The architecture is effectively an output-error feedback controller wrapped around a neural state-space model, so control-theoretic tools for closed-loop stability and disturbance rejection could be applied to predict when the gain matrix will help or hurt.","Because the innovation only requires the target variable to be observed at the current time, the idea should transfer to attention-based sequence models and state-space models such as Mamba, provided they are used in a forecasting setting rather than a pure sequence-generation setting.","A minimal test of the mechanism's contribution would be to train the same IRNN with the innovation input randomly shuffled in time; if gains persist, the effect is not specifically error correction, while if they vanish, the temporal alignment of the error signal is what matters."],"forward_implications":["ILSTM reduces test MSE by roughly 31 percent on ETTh1 and ETTh2, 19 percent on ETTm1, and 27 percent on ETTm2 relative to LSTM, with only 0.73 percent more parameters.","One-step predictions improve by 40 to 60 percent; at the five-step horizon the improvement shrinks to about 10 percent, consistent with the error-correction signal fading as future innovations are zeroed.","IU-BPTT makes the augmentation trainable with off-the-shelf BPTT, at a per-epoch cost of roughly 15 to 20 percent that can be reduced by updating innovations every N epochs with little loss in training MSE.","IRNNs can beat the one-step performance of the naive predictor on datasets where plain RNN, GRU, or LSTM cannot, indicating that innovation feedback helps the model capture dynamics rather than merely interpolating.","The innovation input is a generic extension: applying it to any RNN variant, including gated architectures, yields consistent accuracy gains without changing the training routine qualitatively."],"supporting_citations":[{"why":"Establishes the Kalman filter as the optimal state estimator for linear Gaussian systems, the conceptual source of the innovation idea.","marker":"[21]"},{"why":"Provides a direct data-driven, innovation-based output prediction method for linear systems that the paper generalizes to nonlinear RNNs.","marker":"[24]"},{"why":"Defines the LSTM architecture whose gates and cell update receive the innovation inputs in ILSTM.","marker":"[11]"},{"why":"Defines the GRU architecture that IGRU extends with innovation feedback.","marker":"[12]"},{"why":"Supplies the backpropagation-through-time routine that IU-BPTT invokes for the parameter-update step.","marker":"[42]"},{"why":"Provides the ETT benchmark datasets used for all comparisons in the experiments.","marker":"[46]"},{"why":"Specifies the ADAM optimizer used to train all networks, a standard component of the reported setup.","marker":"[50]"},{"why":"Represents prior work using extended Kalman filtering for RNN training, which the paper contrasts with injecting innovations at the architecture level.","marker":"[36]"}],"fun_headline_variants":["Self-correcting RNNs: feed past errors to slash forecast error","Kalman-style innovation feedback lifts LSTM accuracy by 31%","RNNs adopt Kalman's trick: prediction errors as inputs","Error-driven RNN: 31% lower forecast MSE on ETT benchmarks","Innovation feedback: a new RNN architecture that self-corrects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported multi-step gains rely on the approximation that future prediction errors are zero when the network rolls forward, even though those errors are unknown and typically not zero in practice.","fun_headline_variants_meta":{"raw":{"variants":["Self-correcting RNNs: feed past errors to slash forecast error","Kalman-style innovation feedback lifts LSTM accuracy by 31%","RNNs adopt Kalman's trick: prediction errors as inputs","Error-driven RNN: 31% lower forecast MSE on ETT benchmarks","Innovation feedback: a new RNN architecture that self-corrects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1386,"prompt_tokens":988,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":303}},"tokens_in":604,"tokens_out":398,"duration_ms":4182,"temperature":1.0,"reasoning_tokens":303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:53:14.853838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to train ILSTM under IU-BPTT with the innovation input $e_{t-1}$ replaced by a time-shifted or randomly permuted version of the same prediction error. If test MSE stays as low as with the true $e_{t-1}$, the gain is not the specific Kalman-style feedback; if MSE rises back toward LSTM levels, the immediate one-step error signal is the active component.","supporting_citations":[{"cited_title":"New approach to information fusion steady-state Kalman filtering,","cited_arxiv_id":null,"evidence_quote":"Establishes the Kalman filter as the optimal state estimator for linear Gaussian systems, the conceptual source of the innovation idea."},{"cited_title":"Data-driven output predic- tion and control of stochastic systems: An innovation-based approach,","cited_arxiv_id":null,"evidence_quote":"Provides a direct data-driven, innovation-based output prediction method for linear systems that the paper generalizes to nonlinear RNNs."},{"cited_title":"Backpropagation through time: What it does and how to do it,","cited_arxiv_id":null,"evidence_quote":"Supplies the backpropagation-through-time routine that IU-BPTT invokes for the parameter-update step."},{"cited_title":"Convergence study in extended Kalman filter- based training of recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"Represents prior work using extended Kalman filtering for RNN training, which the paper contrasts with injecting innovations at the architecture level."}],"review_version":1}