Pith. sign in

REVIEW 4 major objections 6 minor 35 references

A Unified Hyperparameter Optimization Pipeline for Transformer-Based Time Series Forecasting Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single hyperparameter-optimization pipeline, built on Optuna and Ray Tune, efficiently finds optimal configurations for transformer-based time series forecasting models and also works for non-transformer models…

desk verdict A workmanlike HPO pipeline paper assembled from existing tools; the pipeline is plausible, but the empirical insights rest on 20 single-seed trials with severe OOM exclusions, so the rankings and cross-model comparisons are not reliable as presented. read the letter →

arxiv 2501.01394 v1 pith:L2LEVG7M submitted 2025-01-02 cs.LG cs.AI

Jingjing Xu , Caesar Wu , Yuan-Fang Li , Grégoire Danoy , Pascal Bouvry This is my paper · ORCID
classification cs.LGcs.AI
keywords hyperparameteroptimizationtimeseriesforecastingtransformermodelsOptunaSearchRayTunebenchmarkMambaMixer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that one reusable hyperparameter-optimization pipeline can efficiently find good hyperparameter settings for transformer-based time series forecasting models. The authors build the pipeline around a tree-structured Parzen estimator search, run 20 trials per model-dataset pair on ETTh1, Weather, and Electricity, and report which hyperparameters matter most. They show that the same pipeline tunes non-transformer state-of-the-art models, Mamba and TimeMixer, as well. If the claim holds, practitioners and researchers can follow the pipeline instead of hand-tuning each model and dataset.

What carries the argument

The load-bearing mechanism is the HPO pipeline itself: it starts from the code base's parameter set, selects common hyperparameters, builds a search space whose lower and upper bounds are one step beyond the minimum and maximum values seen across models, and searches with OptunaSearch, a tree-structured Parzen estimator variant, executed under Ray Tune. The same pipeline logs each trial, removes out-of-memory crashes, and uses parallel-coordinate plots to map hyperparameter values to validation loss, which generates the importance rankings. This is what carries the claim that a single workflow can replace per-model manual tuning.

What would settle it

Re-run the pipeline on the same three datasets with 100 trials per model-dataset pair and several seeds per configuration, and compare the best models and top-three hyperparameter rankings; if TimeMixer no longer leads on ETTh1, Crossformer no longer leads on Weather and ECL, or the importance order changes, the reported conclusions were artifacts of the small sample and exclusions.

Watch

Extended reading notes

Core claim

The central claim is that a unified pipeline—collecting common hyperparameters, defining a search space one step wider than the models' existing ranges, and searching with OptunaSearch under Ray Tune—identifies competitive configurations for four transformer forecasters (Autoformer, Crossformer, Non-Stationary Transformer, PatchTST) and two other state-of-the-art models (Mamba, TimeMixer) on ETTh1, Weather, and ECL. In the reported runs, TimeMixer has the lowest validation error on ETTh1 while Crossformer leads on Weather and ECL. The analysis concludes that model size ($d_{model}$), learning rate, and batch size are the most influential hyperparameters, and that high-dimensional datasets push many trials into out-of-memory failures, so future tuning should constrain model size and batch size or use distributed GPUs.

Load-bearing premise

The load-bearing premise is that twenty tuning attempts per model and dataset, each run once without seed control and with crashed or outlier trials removed, give a reliable picture of each model's best achievable performance and of which hyperparameters matter.

Editorial extensions

If this is right

  • A practitioner can apply the same pipeline to a new transformer-based forecasting model or dataset and get a tuned configuration without designing a bespoke search.
  • Because the pipeline extends to Mamba and TimeMixer, its usefulness is not tied to a particular attention architecture.
  • On datasets with many variables, tuning runs that combine large batch sizes with large model widths will frequently crash, so memory-aware search spaces are a primary constraint.
  • The reported top parameters—$d_{model}$, learning rate, and batch size—give concrete starting points for subsequent tuning rounds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair test would rerun the same pipeline with many more trials and multiple seeds; the paper's 20 single-run trials leave the reported model rankings and parameter importance orders open to sampling noise.
  • The underfitting curves on ETTh1 and Weather suggest the pipeline's chosen epoch range and model-capacity bounds may cap achievable accuracy; extending the search space could change which model wins.
  • The same workflow could be paired with a cost axis—Crossformer's best Weather and ECL results come at much larger training times—to support multi-objective tuning that trades accuracy against compute.
  • Applying the pipeline to prediction lengths 192, 336, and 720 would test whether the importance rankings transfer across forecast horizons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes a unified hyperparameter optimization (HPO) pipeline for transformer-based time series forecasting (TSF), built on OptunaSearch, Ray Tune, and Weights & Biases, and demonstrates it by tuning six models (Autoformer, Crossformer, Non-stationary Transformer, PatchTST, Mamba, TimeMixer) on ETTh1, Weather, and ECL at prediction length 96. It reports the best MSE/MAE per model-dataset pair (Table IV), analyzes out-of-memory (OOM) failures (Tables V-VI), and lists the top-3 hyperparameters per model (Table VII), concluding that the pipeline efficiently identifies optimal configurations and generalizes beyond transformers. The code and full experimental results are released on GitHub.

Significance. If the empirical results are reliable, the paper provides practical guidance for practitioners and a reusable pipeline; strengths include open code and results, standard benchmark datasets, and coverage of six models including two non-transformer baselines. However, the significance is at present conditional: the central quantitative findings rest on only 20 single-seed trials per model-dataset pair, ad hoc exclusion of OOM-crashed and outlying trials, and visually inferred importance rankings, so the reported best models and hyperparameter guidance are not yet established with the confidence the paper claims.

major comments (4)
  1. [Section III-B.2 and Table IV] The HPO budget is only 20 Optuna trials per model-dataset pair, with no seeds or repeated runs. Against the discrete search spaces in Table III (eight hyperparameters, several with ten or more values), 20 trials cannot reliably locate near-optimal configurations, and training stochasticity is uncontrolled. The best-MSE column in Table IV may therefore reflect search luck rather than model quality, so the cross-model rankings (e.g., TimeMixer best on ETTh1, Crossformer best on Weather/ECL) are not load-bearing as reported. Repeating each tuning run with multiple seeds and reporting mean and dispersion (or IQR) is necessary.
  2. [Section IV-B and Tables V-VI] Excluding OOM-crashed trials biases the comparison. On ECL, TimeMixer crashes in 85% of trials, PatchTST in 40%, and Mamba on Weather in 53%, so the 'best' results are often selected from only a few successful configurations (3-15 trials). Since OOM is systematically tied to batch size, d_model, and d_ff (Table VI), the effective search space for high-OOM models is restricted to a memory-feasible subset, meaning the benchmark compares unequal effective search budgets. Please report all trials, including failures, or use a memory-aware search space, and show whether the surviving configurations are representative.
  3. [Section IV-B.3] For Autoformer on Weather, outliers are removed post hoc before deriving parameter importance: 'we remove outliers from Autoformer's results and observed that batch size is the most influential parameter'. No definition, count, or threshold for these outliers is given. This is not reproducible and can change the conclusion; it should be replaced by a pre-specified robust analysis (e.g., median-based ranking or an explicit outlier criterion applied uniformly to all models) or removed.
  4. [Section IV-B.2 and Table VII] The top-3 hyperparameter importance ranking in Table VII is obtained by visual inspection of parallel coordinate plots, not by a quantitative sensitivity measure. With only 20 trials per model-dataset pair (and fewer after OOM exclusions), visual patterns can be driven by a few points, so the rankings are assertions rather than established findings. Please compute a quantitative importance score (e.g., fANOVA, ablation around the best configuration, or permutation importance on a fitted surrogate) and report the number of trials behind each ranking.
minor comments (6)
  1. [Front matter and Section IV-B] There are several typos and formatting issues: '2rd' and '3th' in the author affiliations, 'Hyperparmater' in the Section IV-B heading, and the Section IV-B title should read 'Hyperparameter and Model Metric'.
  2. [Table II] Table II is extremely difficult to read: the column headers mix model names with dataset names, and many cells contain repeated 'DF' values without clear meaning. It should be completely reformatted, with model names as columns and parameters as rows, and 'DF' defined.
  3. [Table III] In Table III, the 'learningrate' row leaves the Lowest and Highest Value columns empty, and 'Searching Space' should be 'Search Space'. Also, the relationship between the listed lowest/highest values and the one-step extension described in Section III-C should be stated explicitly for each parameter.
  4. [Figures 3-6] The captions for Figures 3 and 4 are uninformative: Figure 3 repeats 'ETTh1, Weather, ECL' under every panel, and Figure 4 has only 'Parallel coordinates plot on Weather dataset: Autoformer without outlier'. Please give each panel a descriptive caption explaining what is plotted.
  5. [Abstract and footnote 1] The GitHub URL in the footnote appears as '1https://github.com/jingjing-unilu/HPO transformer time series' with a space and no visible hyperlink formatting. Please provide a working, properly rendered URL.
  6. [Section V and abstract] The claim that the pipeline 'is generalizable beyond transformer-based architectures' is supported only by two non-transformer models (Mamba and TimeMixer) at a single prediction length (96). Please soften the claim or add evidence at other horizons and for additional non-transformer architectures.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper is an empirical HPO benchmark whose comparisons rest on external datasets and model implementations, not on fitted constants or self-cited theorems.

full rationale

The paper's central claim is that its Optuna/Ray Tune based pipeline efficiently identifies hyperparameters for transformer-based TSF models and generalizes to Mamba and TimeMixer. This claim is supported by experiments on ETTh1, Weather, and ECL using standard metrics and published model implementations. No derivation in the paper takes a fitted quantity and renames it a prediction; the reported 'best model' is simply the minimum validation MSE observed over the executed trials. The search space is defined by extending the min/max parameter values across models, which is a design convention rather than a circular reduction. The only self-citations are [30], [31], [26], and [27], and they appear in peripheral roles: [30] is cited as a survey for metrics and general background, [31] is cited for the generic suggestion that feature engineering may help underfitting, and [26]/[27] are cited as examples of transformer-based forecasting in finance. None of these citations supplies a load-bearing premise, uniqueness theorem, or ansatz that the paper's conclusions depend on. The main threats to the empirical conclusions—small trial counts, OOM-driven exclusions, and outlier removal—concern statistical reliability and experimental validity, not circularity, and therefore do not raise the circularity score. The paper is self-contained against external benchmarks and exhibits no self-definitional or fitted-input-as-prediction pattern.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on the reliability of the benchmark protocol rather than on mathematical axioms. There are no free parameters fitted to data in the sense of derived constants; the manually chosen search spaces and the 20-trial-per-run protocol are load-bearing assumptions. No new entities are introduced.

assumptions (3)
  • ad hoc to paper The manually constructed search spaces in Tab. III (lower bound one step below the minimum across models, upper bound one step above the maximum) are adequate to cover near-optimal hyperparameters for every model.
    Introduced in Sec. III-C.1; if the true optima lie outside these ranges, the best results and importance rankings are invalid.
  • domain assumption 20 tuning trials per model-dataset pair, executed once without repeated seeds, are enough to approximate the best achievable validation MSE.
    Sec. III-B.2 states 20 trials per dataset; no variance or seed control is reported, so the best-of-20 estimate's stability is assumed.
  • domain assumption Trials that fail with out-of-memory errors and declared outliers are missing at random and do not bias the comparison.
    Sec. IV-B reports OOM rates up to 85% and, for Autoformer on Weather, explicitly removes outliers before ranking parameters; the analysis assumes these exclusions are benign.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Unified Hyperparameter Optimization Pipeline for Transformer-Based Time Series Forecasting Models." pith.science (2026). https://pith.science/paper/L2LEVG7M

@misc{pith2026250101394,
  author       = {Pith},
  title        = {Pith review of: A Unified Hyperparameter Optimization Pipeline for Transformer-Based Time Series Forecasting Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2LEVG7M}},
  note         = {Machine review of arXiv:2501.01394}
}
read the original abstract

Transformer-based models for time series forecasting (TSF) have attracted significant attention in recent years due to their effectiveness and versatility. However, these models often require extensive hyperparameter optimization (HPO) to achieve the best possible performance, and a unified pipeline for HPO in transformer-based TSF remains lacking. In this paper, we present one such pipeline and conduct extensive experiments on several state-of-the-art (SOTA) transformer-based TSF models. These experiments are conducted on standard benchmark datasets to evaluate and compare the performance of different models, generating practical insights and examples. Our pipeline is generalizable beyond transformer-based architectures and can be applied to other SOTA models, such as Mamba and TimeMixer, as demonstrated in our experiments. The goal of this work is to provide valuable guidance to both industry practitioners and academic researchers in efficiently identifying optimal hyperparameters suited to their specific domain applications. The code and complete experimental results are available on GitHub.

Figures

Figures reproduced from arXiv: 2501.01394 by the authors.

Figure 1
Figure 1. HPO pipeline for Transformer-based forecasting [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Common Parameters in the Transformer while underline numbers denote the second-best performance. For the ETTh1 dataset, TimeMixer achieved the best results. In contrast, for the Weather and ECL datasets, Crossformer demonstrates better performance even it is a time-consuming model. The learning curve in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training loss and validation loss on each model’s best performance case in experiments [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Parallel coordinates plot on Weather dataset: Autoformer [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Parallel coordinates plot on ETTh1 dataset [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 31 canonical work pages

  1. [1]

    Mar. 2024. URL: https : / / www . geeksforgeeks . org / underfitting-and-overfitting-in-machine-learning/

    work page 2024

  2. [2]

    Transformers in time-series anal- ysis: A tutorial

    Sabeen Ahmed et al. “Transformers in time-series anal- ysis: A tutorial”. In: Circuits, Systems, and Signal Pro- cessing 42.12 (2023), pp. 7433–7466

    work page 2023

  3. [3]

    Neural architecture search benchmarks: Insights and survey

    Krishna Teja Chitty-Venkata et al. “Neural architecture search benchmarks: Insights and survey”. In: IEEE Ac- cess 11 (2023), pp. 25217–25236

    work page 2023

  4. [4]

    Neural architecture search for transformers: A survey

    Krishna Teja Chitty-Venkata et al. “Neural architecture search for transformers: A survey”. In: IEEE Access 10 (2022), pp. 108374–108412

    work page 2022

  5. [5]

    Hyperparame- ter search in machine learning

    Marc Claesen and Bart De Moor. “Hyperparame- ter search in machine learning”. In: arXiv preprint arXiv:1502.02127 (2015)

    arXiv 2015

  6. [6]

    25 years of time series forecasting

    Jan G De Gooijer and Rob J Hyndman. “25 years of time series forecasting”. In: International journal of forecasting 22.3 (2006), pp. 443–473

    work page 2006

  7. [7]

    Algorithms for hyperparameter tuning of lstms for time series forecasting

    Harshal Dhake, Yashwant Kashyap, and Panagiotis Kos- mopoulos. “Algorithms for hyperparameter tuning of lstms for time series forecasting”. In: Remote Sensing 15.8 (2023), p. 2076

    work page 2023

  8. [8]

    Hyperparameter op- timization

    Matthias Feurer and Frank Hutter. “Hyperparameter op- timization”. In: Automated machine learning: Methods, systems, challenges (2019), pp. 3–33

    work page 2019

Show all 35 references
  1. [9]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. “Mamba: Linear-time sequence modeling with selective state spaces”. In: arXiv preprint arXiv:2312.00752 (2023)

    2023 arXiv

  2. [10]

    Advanced hyperparameter optimization of deep learning models for wind power prediction

    Shahram Hanifi, Andrea Cammarono, and Hossein Zare- Behtash. “Advanced hyperparameter optimization of deep learning models for wind power prediction”. In: Renewable Energy 221 (2024), p. 119700

    2024

  3. [11]

    Multivariate time series forecasting with dynamic graph neural odes

    Ming Jin et al. “Multivariate time series forecasting with dynamic graph neural odes”. In: IEEE Transac- tions on Knowledge and Data Engineering 35.9 (2022), pp. 9168–9180

    2022

  4. [12]

    Time-LLM: Time Series Forecasting by Reprogramming Large Language Models

    Ming Jin et al. “Time-LLM: Time Series Forecasting by Reprogramming Large Language Models”. In: ICLR. 2024

    2024

  5. [13]

    iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

    Yong Liu et al. “iTransformer: Inverted Transformers Are Effective for Time Series Forecasting”. In: ICLR. 2024

    2024

  6. [14]

    Non-stationary transformers: Exploring the stationarity in time series forecasting

    Yong Liu et al. “Non-stationary transformers: Exploring the stationarity in time series forecasting”. In: NeurIPS (2022)

    2022

  7. [15]

    Review of automated time series forecasting pipelines

    Stefan Meisenbacher et al. “Review of automated time series forecasting pipelines”. In: Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12.6 (2022), e1475

    2022

  8. [16]

    A Time Series is Worth 64 Words: Long- term Forecasting with Transformers

    Yuqi Nie et al. “A Time Series is Worth 64 Words: Long- term Forecasting with Transformers”. In: ICLR. 2023

    2023

  9. [17]

    Forecasting: theory and prac- tice

    Fotios Petropoulos et al. “Forecasting: theory and prac- tice”. In: International Journal of Forecasting 38.3 (2022), pp. 705–871

    2022

  10. [18]

    Training tips for the transformer model

    Martin Popel and Ond ˇrej Bojar. “Training tips for the transformer model”. In: arXiv preprint arXiv:1804.00247 (2018)

    2018 arXiv

  11. [19]

    Autotransformer: Automatic trans- former architecture design for time series classification

    Yankun Ren et al. “Autotransformer: Automatic trans- former architecture design for time series classification”. In: PAKDD. Springer. 2022, pp. 143–155

    2022

  12. [20]

    Distributed hyperparameter opti- mization based multivariate time series forecasting

    Tinku Singh et al. “Distributed hyperparameter opti- mization based multivariate time series forecasting”. In: Multimedia Tools and Applications 83.2 (2024), pp. 5031–5053

    2024

  13. [21]

    Hypertuned temporal fusion transformer for multi-horizon time series fore- casting of dam level in hydroelectric power plants

    Stefano Frizzo Stefenon et al. “Hypertuned temporal fusion transformer for multi-horizon time series fore- casting of dam level in hydroelectric power plants”. In: International Journal of Electrical Power & Energy Systems 157 (2024), p. 109876

    2024

  14. [22]

    Optimization of deep neural networks: a survey and unified taxonomy

    El-Ghazali Talbi. “Optimization of deep neural networks: a survey and unified taxonomy”. working paper or preprint. June 2020. URL: https://inria.hal.science/hal- 02570804

    2020

  15. [23]

    TimeMixer: Decomposable Multi- scale Mixing for Time Series Forecasting

    Shiyu Wang et al. “TimeMixer: Decomposable Multi- scale Mixing for Time Series Forecasting”. In: ICLR. 2024

    2024

  16. [24]

    Time Series Data Augmentation for Deep Learning: A Survey

    Qingsong Wen et al. “Time Series Data Augmentation for Deep Learning: A Survey”. In: IJCAI. 2021

    2021

  17. [25]

    Transformers in time series: a survey

    Qingsong Wen et al. “Transformers in time series: a survey”. In: IJCAI. 2023

    2023

  18. [26]

    Strategic Predictions and Explana- tions By Machine Learning

    Caesar WU et al. “Strategic Predictions and Explana- tions By Machine Learning”. In: The 38th International Conference on Information Networking . 2024

    2024

  19. [27]

    Trustworthy AI: Deciding What to Decide

    Caesar WU et al. “Trustworthy AI: Deciding What to Decide”. In: 12th Computing Conf. 2024

    2024

  20. [28]

    Autoformer: decomposition transform- ers with auto-correlation for long-term series forecast- ing

    Haixu Wu et al. “Autoformer: decomposition transform- ers with auto-correlation for long-term series forecast- ing”. In: Proc. 35th Int. Conf. NeurIPS. 2021, pp. 22419– 22430

    2021

  21. [29]

    TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis

    Haixu Wu et al. “TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis”. In: ICLR. 2023

    2023

  22. [30]

    Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Fore- casting

    Jingjing Xu et al. “Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Fore- casting”. In: arXiv preprint arXiv:2407.19784 (2024)

    2024 arXiv

  23. [31]

    Transformer Multivariate Forecasting: Less is More?

    Jingjing Xu et al. “Transformer Multivariate Forecasting: Less is More?” In: AI4TS@AAAI. 2024

    2024

  24. [32]

    On hyperparameter op- timization of machine learning algorithms: Theory and practice

    Li Yang and Abdallah Shami. “On hyperparameter op- timization of machine learning algorithms: Theory and practice”. In: Neurocomputing 415 (2020), pp. 295–316

    2020

  25. [33]

    Hyper-parameter optimization: A review of algorithms and applications

    Tong Yu and Hong Zhu. “Hyper-parameter optimization: A review of algorithms and applications”. In: arXiv preprint arXiv:2003.05689 (2020)

    2020 arXiv

  26. [34]

    Crossformer: Trans- former utilizing cross-dimension dependency for multi- variate time series forecasting

    Yunhao Zhang and Junchi Yan. “Crossformer: Trans- former utilizing cross-dimension dependency for multi- variate time series forecasting”. In: ICLR. 2022

    2022

  27. [35]

    Informer: Beyond efficient trans- former for long sequence time-series forecasting

    Haoyi Zhou et al. “Informer: Beyond efficient trans- former for long sequence time-series forecasting”. In: Proc. AAAI conf. V ol. 35. 12. 2021, pp. 11106–11115

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.