REVIEW 5 major objections 7 minor 39 references
Human in the Loop Adaptive Optimization for Improved Time Series Forecasting
T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper proves that a validation-fitted affine correction always reduces the mean squared error of a time series forecast, then builds a post-training action-selection framework around that guarantee.
desk verdict A sound but textbook theory, a promising LLM-to-action idea, and an experimental section whose own numbers contradict the paper's central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the affine correction identity, $Y_{\mathrm{corrected}}=aY_{\mathrm{pred}}+b$ with $a,b$ fitted by least squares on the validation set; the proof rewrites the risk gap as $\left(\sqrt{\mathrm{Var}(Y_{\mathrm{pred}})} - \frac{\mathrm{Cov}(Y_{\mathrm{true}},Y_{\mathrm{pred}})}{\sqrt{\mathrm{Var}(Y_{\mathrm{pred}})}}\right)^2$, a square, so it is always nonnegative. Around that identity the paper builds a hybrid action space of discrete transformations, such as scale amplitude, piecewise quantile scaling, linear trend, min/max adjustment, shift, and noise, each with continuous parameters, plus a search-and-select loop that picks the action and parameters minimizing validation MSE. The human-in-the-loop component adds a language model that translates free-text feedback into executable action code, which is then fed into the same selection loop.
What would settle it
Run the paper's own procedure on a fixed dataset and model while splitting the historical data into many validation/test pairs; if the action selected on validation reduces test MSE less than half the time, or if re-running the published Table 2 settings reproduces negative entries such as PatchTST on ETTh1 (-2.25%) or Crossformer on Nature (-1.17%), then the claimed consistent gains do not transfer as stated. A direct calculation would also suffice: fit the affine correction on one window and evaluate the guaranteed gap on a later window with a different mean or variance.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that post-training correction is not just a heuristic: for a predictor $Y_{\mathrm{pred}}$ and a target $Y_{\mathrm{true}}$, setting $a^* = \mathrm{Cov}(Y_{\mathrm{true}},Y_{\mathrm{pred}})/\mathrm{Var}(Y_{\mathrm{pred}})$ and $b^* = \mathbb{E}[Y_{\mathrm{true}}]-a^*\mathbb{E}[Y_{\mathrm{pred}}]$ makes $Y_{\mathrm{corrected}}=a^*Y_{\mathrm{pred}}+b^*$ satisfy $\mathrm{MSE}_{\mathrm{before}}-\mathrm{MSE}_{\mathrm{after}} \ge 0$, with the gap expressed as an exact square. The paper claims this guarantee motivates a general action-search layer: a discrete set of interpretable transformations with continuous parameters, optimized by random search, bandits, reinforcement learning, or genetic algorithms, that consistently improves forecasts across multiple benchmarks and base models without retraining. It further claims that expert feedback expressed in natural language can be parsed into such actions and produce additional gains.
Load-bearing premise
The load-bearing premise is that the validation set used to choose and fit the correction has the same error structure as the test set; if the distribution shifts between validation and test, the chosen action can degrade rather than improve the forecasts.
Editorial extensions
If this is right
- Any deployed forecaster can be given a cheap calibration layer: fit $a,b$ on validation data and apply the affine correction, with no retraining and no architecture change.
- Because the risk gap is an exact square, the affine correction is safe on the validation distribution; the danger is entirely in transfer to a different test distribution.
- The richer action pool can produce larger gains than the affine fix, including double-digit MSE reductions in the paper's tables, at the price of losing the mathematical guarantee.
- SH-HPO and random search are reported as the most consistent optimizers, while PPO and genetic search trail on these benchmarks, suggesting the search strategy is a real design choice.
- Natural-language feedback can be absorbed as extra candidate actions, making the same loop usable by domain experts who do not write code.
Reading between the lines
- A testable extension would compare the richer action search against the affine-only baseline on every dataset; the reported gains that exceed the affine gap measure what the extra transformations actually buy.
- The validation-selection procedure is itself a model-selection step, so as the action pool grows, overfitting to validation noise should eventually flip gains negative; the negative entries in the paper's tables are consistent with that effect.
- The language-model-to-action parsing could be ablated by having a human hand-code the same instructions; the difference would separate parsing quality from the optimizer's contribution.
- For deployment, a practical safeguard is to choose actions on rolling validation windows rather than one fixed split, and to fall back to the affine correction when a richer action does not beat it on validation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a post-training optimization framework for time series forecasting that applies interpretable transformations to the outputs of any base forecaster, with an optional human-in-the-loop component in which natural-language feedback is parsed by an LLM into executable corrections. The authors prove a correct theorem showing that an optimal affine correction never increases MSE on the distribution used to fit it, then describe random search, bandit (SH-HPO), RL (PPO), and genetic-algorithm variants over a broader action space. The main empirical claim, presented in Table 2, is that this post-training search yields consistent test-set MSE reductions across seven models and ten datasets. The paper also reports optimization-time measurements and qualitative human-feedback case studies.
Significance. If the empirical claims were substantiated, the framework would be a genuinely lightweight, model-agnostic add-on to forecasting pipelines, and the released code and interactive demo would support deployment and reproducibility. The affine-correction derivation (Theorem 1 and Appendix 6.8) is self-contained, correct for the stated setting, and a useful elementary observation. However, the central evidence for the framework's practical value is Table 2, and that table is not internally consistent: several cells contain duplicated values across independent model-dataset pairs, and many reported percentage improvements cannot be derived from the printed before/after numbers. The paper's own data also contains negative improvements that contradict the abstract's wording of 'consistent accuracy gains.' As presented, the empirical contribution is unreliable and needs to be redone before the paper can be evaluated on its merits.
major comments (5)
- [Section 4.2, Table 2] Table 2 cannot be used as a record of the experiments. In the ETTh2 row, all seven models are reported as 0.60±0.01 → 0.57±0.01, yet the parenthetical improvements are 15.48%, 3.7%, 4.0%, 4.2%, 3.8%, 3.8%, and 3.9%, none of which equals the arithmetic improvement of 5.0% implied by 0.60 → 0.57. In the ETTm2 row, Crossformer shows 3.82±0.01 → 3.81±0.01, an improvement of about 0.26%, but is labeled 20.25%. These are not rounding artifacts. The averaged rows in Appendix 6.3.2 also do not match the per-horizon rows above them; for example, Crossformer on ETTh2 has per-horizon before-values of 1.10, 1.59, 6.57, and 0.55, but the 'averaged' row in the appendix and the corresponding cell in Table 2 both display 0.60±0.01. Because this table is the primary quantitative support for the claim of consistent gains, the empirical claim is currently unsupported.
- [Section 3.1, Theorem 1 and Eq. (20)] The theorem is correct but is narrower than the framework it is used to justify. It guarantees non-increase of MSE only for the affine correction with parameters fitted on the same distribution on which the MSE is evaluated. It does not cover the piecewise scaling, noise addition, shift, or min/max adjustment actions defined in Section 3.3 and Table 4, nor does it provide a test-set guarantee under distribution shift. The paper's abstract and introduction state 'we prove that affine corrections always reduce the mean squared error' and then present the whole action-search framework as an extension; as written, this conflates the theorem's guarantee with the unproven behavior of the richer action space. The authors should clearly separate what is proved from what is hypothesized, and the empirical evaluation must stand on its own.
- [Section 4.2 and Appendix 6.3.2] The claim of 'consistent accuracy gains' is contradicted by entries in the paper's own tables. In Table 2, PatchTST on ETTh1 is reported at -2.25%, Crossformer on KDD at -0.03%, Crossformer on Nature at -1.17%, and Crossformer on NASDAQ at -0.33%. Appendix 6.3.2 contains additional negatives, such as SegRNN on ETTh2 at horizon 192 (-5.88%) and Autoformer on Tourism at horizon 336 (-3.60%). If the intended claim is that gains occur in a majority of cases rather than universally, the wording must be weakened and the actual win-rate reported. As it stands, the abstract and Section 5 assert consistency that the data do not support.
- [Section 3.3.3 and Section 4.1] The protocol for selecting actions on a validation set and then reporting test MSE is not protected against selection overfitting. The paper states that actions are evaluated globally on the validation set and that consistency is checked on the training set, but no results of that consistency check are given, no nested or repeated evaluation is described, and the reported standard deviations (often ±0.01) are not accompanied by any significance testing or confidence intervals. Given that the action space has several continuous parameters and the search evaluates many candidate transformations, the observed test-set improvements need to be validated against the selection procedure itself. Please provide per-horizon, per-seed results and a corrected metric (e.g., validation-selected actions evaluated on a genuinely held-out test set, with error bars that reflect the selection).
- [Section 4.2.1 and Appendix 6.4.1] The human-in-the-loop contribution is evaluated only qualitatively. Figure 6 and the three appendix case studies show that LLM-parsed feedback can change forecasts and that RMSE improves in these examples, but there is no systematic comparison between automated optimization alone and automated optimization with human feedback across models and datasets. Without a quantitative protocol (e.g., defined feedback tasks, multiple users, measured improvement over the automated baseline), the claimed benefit of the HITL component is not supported. This is a major gap for a framework whose novelty is explicitly the combination of automated post-hoc optimization and human feedback.
minor comments (7)
- [Section 4.1] The forecast horizons are listed as '96, 192, 336 and 792'; the standard long-term forecasting horizons in this benchmark family are 96, 192, 336, and 720, and Table 3 itself uses 720. Please correct the typo.
- [Table 1 caption and Section 3.4] Table 1 is captioned 'averaged over 10 trials on the Nature dataset,' but Section 3.4 states the comparison is performed on the ETTh1 dataset, and the overall averages in Table 1 differ from the 'Overall Average' row in Table 6 (e.g., 4.96% vs. 5.65% for SH-HPO). The dataset and the source of each average should be clarified.
- [Section 6.1, Table 4] In Table 4, 'Piecewise Scale High' is defined as applying y = x·(1+f/100) for x_t ≤ Q_δ and 'Piecewise Scale Low' applies for x_t > Q_δ; this appears to reverse the intended high/low semantics. Please check the definitions and their use in the code.
- [Appendix 6.8.4] In the derivation of R_before − R_after, the text says 'by setting x=Var(Y_pred) and a=Cov(Y_true, Y_pred)^2'; the identity only holds with a = Cov(Y_true, Y_pred), as the final expression shows. Please fix this intermediate variable assignment.
- [Multiple figures] The paper uses both MSE and RMSE inconsistently: Section 4 reports MSE as the primary metric, but Figure 6 and Appendix 6.4.1 describe 'RMSE improvement.' Please standardize the metric to avoid confusion.
- [Figure 15 caption] The caption contains a typo: 'Successfull' should be 'Successful.'
- [Appendix 6.6-6.7] The command-line example '--train_path <path_to_train_data' is missing a closing angle bracket, and the GitHub repository is referenced only by URL; consider adding a persistent archival DOI or a versioned release for reproducibility.
Circularity Check
No significant circularity: the theoretical result is a genuine derivation and the empirical claims rest on standard held-out evaluation.
full rationale
The paper's derivation chain is self-contained. Theorem 1 fixes a* and b* as the minimizers of the MSE objective L(a,b) = E[||Ytrue - (aYpred + b)||^2] (Eqs. 9-11), then computes Rbefore - Rafter by direct algebra (Eq. 20). The non-negativity follows from completing the square and does not assume the conclusion; it is a mathematical identity about the optimal affine correction on the distribution used to fit it. The paper explicitly conditions the transfer guarantee on a distributional assumption: 'This guarantee holds as long as the validation and test data are drawn from the same distribution.' The later action-search framework (random search, bandits, PPO, GA) is optimized on a validation split and then reported on test data; this is standard held-out evaluation, not a fitted parameter being renamed as a prediction. The abstract's phrase 'Across nearly all models and datasets, we observe consistent accuracy gains' is an empirical summary of Table 2, and while several cells in Table 2 are internally inconsistent and even negative (e.g., PatchTST on ETTh1 shows -2.25%, Crossformer on Nature shows -1.17%), contradiction with the table is a correctness or reproducibility risk, not circularity. No load-bearing self-citation is present: the cited prior work supplies models, datasets, and optimization algorithms, and no uniqueness theorem is imported from the authors' own prior work to force the framework's choices. I therefore find no circular step that reduces the paper's claims to their own inputs.
Assumptions & free parameters
free parameters (1)
- action parameters (scale factor f, quantile threshold delta, shift delta, slope s, intercept b, noise sigma) =
tuned via random search or line search on the validation set per dataset
assumptions (4)
- domain assumption Validation and test data are drawn from the same distribution
- ad hoc to paper A small set of hand-chosen transformations (scale, shift, trend, quantile adjustments) is expressive enough to capture the base model's systematic errors
- domain assumption The LLM (qwen2-72b-32k) reliably maps natural language feedback to correct executable transformations
- standard math Least-squares projection property of the best linear predictor
Cite this review
Pith. "Pith review of Human in the Loop Adaptive Optimization for Improved Time Series Forecasting." pith.science (2026). https://pith.science/paper/OY6BHOIG
@misc{pith2026250515354,
author = {Pith},
title = {Pith review of: Human in the Loop Adaptive Optimization for Improved Time Series Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/OY6BHOIG}},
note = {Machine review of arXiv:2505.15354}
}
read the original abstract
Time series forecasting models often produce systematic, predictable errors even in critical domains such as energy, finance, and healthcare. We introduce a novel post training adaptive optimization framework that improves forecast accuracy without retraining or architectural changes. Our method automatically applies expressive transformations optimized via reinforcement learning, contextual bandits, or genetic algorithms to correct model outputs in a lightweight and model agnostic way. Theoretically, we prove that affine corrections always reduce the mean squared error; practically, we extend this idea with dynamic action based optimization. The framework also supports an optional human in the loop component: domain experts can guide corrections using natural language, which is parsed into actions by a language model. Across multiple benchmarks (e.g., electricity, weather, traffic), we observe consistent accuracy gains with minimal computational overhead. Our interactive demo shows the framework's real time usability. By combining automated post hoc refinement with interpretable and extensible mechanisms, our approach offers a powerful new direction for practical forecasting systems.
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[1]
F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S
Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. (2024). Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815
arXiv 2024
-
[2]
Armstrong, J. S. (1986). The ombudsman: research on forecasting: A quarter-century review, 1960--1984. Interfaces , 16(1):89--109
work page 1986
-
[3]
Arvan, M., Fahimnia, B., Reisi, M., and Siemsen, E. (2019). Integrating human judgement into quantitative forecasting methods: A review. Omega , 86:237--252
work page 2019
-
[4]
Bunn, D. and Wright, G. (1991). Interaction of judgemental and statistical forecasting methods: issues & analysis. Management science , 37(5):501--518
work page 1991
-
[5]
Das, A., Kong, W., Sen, R., and Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning
work page 2024
-
[6]
Gardner Jr, E. S. (1985). Exponential smoothing: The state of the art. Journal of forecasting , 4(1):1--28
work page 1985
-
[7]
Geweke, J. and Whiteman, C. (2006). Bayesian forecasting. Handbook of economic forecasting , 1:3--80
work page 2006
-
[8]
Girard, A., Rasmussen, C., Candela, J. Q., and Murray-Smith, R. (2002). Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting. Advances in neural information processing systems , 15
work page 2002
Show all 39 references
-
[9]
and Graves, A
Graves, A. and Graves, A. (2012). Long short-term memory. Supervised sequence labelling with recurrent neural networks , pages 37--45
2012
-
[10]
Holland, J. H. (1975). Adaptation in Natural and Artificial Systems . University of Michigan Press, Ann Arbor, MI
1975
-
[11]
Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., and Redko, I. (2024). Samformer: Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention. arXiv preprint arXiv:2402.10198
2024 arXiv
-
[12]
S., Koren, T., and Somekh, O
Karnin, Z. S., Koren, T., and Somekh, O. (2013). Almost optimal exploration in multi-armed bandits. In Proceedings of the 30th International Conference on Machine Learning , pages 1238--1246. PMLR
2013
-
[13]
K., Dasgupta, N., Natarajan, S., Pickett, L
Kaushik, S., Choudhury, A., Sheron, P. K., Dasgupta, N., Natarajan, S., Pickett, L. A., and Dutt, V. (2020). Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures. Frontiers in big data , 3:4
2020
-
[14]
and Bijari, M
Khashei, M. and Bijari, M. (2012). A new class of hybrid models for time series forecasting. Expert Systems with Applications , 39(4):4344--4357
2012
-
[15]
Korstanje, J. (2021). The sarima model. In Advanced Forecasting with Python: With State-of-the-Art-Models Including LSTMs, Facebook’s Prophet, and Amazon’s DeepAR , pages 115--122. Springer
2021
-
[16]
Krollner, B., Vanstone, B., and Finnie, G. (2010). Financial time series forecasting with machine learning techniques: A survey. In European Symposium on Artificial Neural Networks: Computational Intelligence and Machine Learning , pages 25--30
2010
-
[17]
L., Skorburg, J
Kupferschmidt, K. L., Skorburg, J. G., and Taylor, G. W. (2022). Delphai: A human-centered approach to time-series forecasting. In 2022 IEEE International Conference on Big Data (Big Data) , pages 4014--4020. IEEE
2022
-
[18]
Lin, S., Lin, W., Wu, W., Zhao, F., Mo, R., and Zhang, H. (2023). Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200
2023 arXiv
-
[19]
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024a). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437
2024 arXiv
-
[20]
B., Sharma, M., Cui, J., Wen, Q., Zhang, C., et al
Liu, H., Xu, S., Zhao, Z., Kong, L., Kamarthi, H., Sasanur, A. B., Sharma, M., Cui, J., Wen, Q., Zhang, C., et al. (2024b). Time-mmd: A new multi-domain multimodal dataset for time series analysis. arXiv preprint arXiv:2406.08627
2024 arXiv
-
[21]
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. (2023). itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625
2023 arXiv
-
[22]
Madadgar, S., Moradkhani, H., and Garen, D. (2014). Towards improved post-processing of hydrologic forecast ensembles. Hydrological Processes , 28(1):104--122
2014
-
[23]
a tz, M., M \
Meisenbacher, S., Turowski, M., Phipps, K., R \"a tz, M., M \"u ller, D., Hagenmeyer, V., and Mikut, R. (2022). Review of automated time series forecasting pipelines. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 12(6):e1475
2022
-
[24]
Newbold, P. (1983). Arima model building and the time series analysis approach to forecasting. Journal of forecasting , 2(1):23--35
1983
-
[25]
Nguyen, N., Sinthong, P., and Kalagnanam, J
Nie, Y., H. Nguyen, N., Sinthong, P., and Kalagnanam, J. (2023). A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations
2023
-
[26]
Palma, G., Chengalipunath, E. S. J., and Rizzo, A. (2024). Time series forecasting for energy management: Neural circuit policies (ncps) vs. long short-term memory (lstm) networks. Electronics , 13(18):3641
2024
-
[27]
Qi, Y., Hu, H., Lei, D., Zhang, J., Shi, Z., Huang, Y., Chen, Z., Lin, X., and Shen, Z.-J. M. (2025). Timehf: Billion-scale time series models guided by human feedback. arXiv preprint arXiv:2501.15942
2025 arXiv
-
[28]
S., Sheng, Z., and Yang, B
Qiu, X., Hu, J., Zhou, L., Wu, X., Du, J., Zhang, B., Guo, C., Zhou, A., Jensen, C. S., Sheng, Z., and Yang, B. (2024). Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods. Proc. VLDB Endow. , 17:2363 -- 2377
2024
-
[29]
R., Ghonia, H., Bhagwatkar, R., Khorasani, A., Bayazi, M
Rasul, K., Ashok, A., Williams, A. R., Ghonia, H., Bhagwatkar, R., Khorasani, A., Bayazi, M. J. D., Adamopoulos, G., Riachi, R., Hassen, N., et al. (2023). Lag-llama: Towards foundation models for probabilistic time series forecasting. arXiv preprint arXiv:2310.08278
2023 arXiv
-
[30]
Sampson, J. R. (1976). Adaptation in natural and artificial systems (john h. holland)
1976
-
[31]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
2017 arXiv
-
[32]
Tavenard, R., Faouzi, J., Vandewiele, G., Divo, F., Androz, G., Holtz, C., Payne, M., Yurchak, R., Ru wurm, M., Kolar, K., et al. (2020). Tslearn, a machine learning toolkit for time series data. Journal of machine learning research , 21(118):1--6
2020
-
[33]
Verkade, J., Brown, J., Reggiani, P., and Weerts, A. (2013). Post-processing ecmwf precipitation and temperature ensemble reforecasts for operational hydrologic forecasting at various spatial scales. Journal of Hydrology , 501:73--91
2013
-
[34]
and O'Connor, M
Webby, R. and O'Connor, M. (1996). Judgemental and statistical time series forecasting: a review of the literature. International Journal of forecasting , 12(1):91--118
1996
-
[35]
Wu, H., Xu, J., Wang, J., and Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419--22430
2021
-
[36]
Zeng, A., Chen, M., Zhang, L., and Xu, Q. (2023). Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121--11128
2023
-
[37]
and Yan, J
Zhang, Y. and Yan, J. (2023). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations
2023
-
[38]
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106--11115
2021
-
[39]
Zhou, Y. (2020). Real-time probabilistic forecasting of river water quality under data missing situation: Deep learning plus post-processing techniques. Journal of Hydrology , 589:125164
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.