{"id":"dc2d250e-8178-4b15-9794-208cbd53ea34","arxiv_id":"2501.01394","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A unified HPO pipeline built on Optuna and Ray Tune is applied to six time series forecasting models across three datasets, providing empirical guidance on hyperparameter choices.","lead":"This paper proposes a unified hyperparameter optimization pipeline for transformer-based time series forecasting models and benchmarks six models on three standard datasets. It reports which settings work best and which hyperparameters matter most, aiming to help practitioners tune forecasting models more efficiently.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's central empirical conclusions rest on 20 single-seed trials with up to 85% OOM-driven exclusions; without replication the reported best models and hyperparameter rankings may be noise.","rationale":"The paper proposes a practical HPO pipeline built on standard tools (Optuna, Ray Tune, Weights & Biases) and tests it on six models and three datasets. I read this as a systems/benchmark contribution rather than a new theoretical method, and the pipeline logic is coherent. The load-bearing premise is empirical: the benchmark results in Table IV and the parameter importance rankings in Table VII are the evidence for the central claim that the pipeline can efficiently identify optimal hyperparameters and provide generalizable insights. That premise requires the results to be stable across the stochasticity of TPE and to be comparable across models. The paper fixes 20 trials per model-dataset pair, does not mention random seeds, removes OOM-crashed trials, and reports OOM rates up to 85%; for some entries fewer than a handful of successful trials remain. Additionally, OOM failures are not missing at random—they depend on batch size and model dimensions—so the surviving trials define a restricted search subspace for high-OOM models. This makes the cross-model comparison and the parameter importance lists potentially noisy or biased. The reader's weakest assumption identified the same issue, and I agree with that reading. I do not see a more fundamental flaw: the pipeline's components are standard and the experiments, while weak, do not contradict the possibility that the pipeline works. The appropriate disposition remains conditional acceptance pending a robustness check; hence I recommend no change to the reader's verdict. The concrete replication test with multiple seeds and more trials would settle whether the reported conclusions are reproducible or artifacts of the small, censored sample.","tokens_in":8824,"tokens_out":5865,"duration_ms":61711,"concrete_test":"Run a controlled replication on ECL for at least PatchTST, Crossformer, and TimeMixer: 5 seeds × 50 trials (or 100 if feasible) per model, keeping the other settings in Sec. III fixed, and record best validation MSE and top-3 hyperparameters for each seed. If the best model in Table IV changes across seeds or the Table VII rankings replicate in fewer than 4 of 5 seeds, the reported conclusions are artifacts of the small single-seed sample; if winners and rankings are stable, the concern does not land. Also report the number of successful trials per model so comparisons are made on equal footing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract, Sec. IV) is that the pipeline efficiently identifies optimal hyperparameters and that the resulting benchmark—best model per dataset and the top-3 parameter importance list in Table VII—is a reliable practical insight. For that claim to hold, the best validation MSE after HPO and the observed parameter sensitivities must be stable under search stochasticity and comparable across models. This condition is not supported. Sec. III-B.2 fixes 20 Optuna trials per model-dataset pair with no seed control. Sec. IV-B then excludes OOM-crashed trials and, for Autoformer on Weather, removes unspecified outliers; OOM rates reach 85% (TimeMixer on ECL), 53% (Mamba on Weather), and 40% for several ECL entries, so some 'best' results are selected from as few as 3–13 successful trials. Since OOM failures are systematically tied to batch size, d_model, and d_ff (Table VI), models with high OOM rates are effectively searched over a restricted memory-feasible subspace; the cross-model comparison in Table IV is therefore not a comparison of each model's best achievable performance under the same search budget. With 8 hyperparameters over the discrete spaces of Table III, 20 (or fewer) points are also far too sparse for the visual importance rankings in Table VII. The pipeline itself may still be sound; the empirical demonstration of the central claim is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a unified hyperparameter optimization (HPO) pipeline for transformer-based time series forecasting (TSF), built on OptunaSearch, Ray Tune, and Weights & Biases, and demonstrates it by tuning six models (Autoformer, Crossformer, Non-stationary Transformer, PatchTST, Mamba, TimeMixer) on ETTh1, Weather, and ECL at prediction length 96. It reports the best MSE/MAE per model-dataset pair (Table IV), analyzes out-of-memory (OOM) failures (Tables V-VI), and lists the top-3 hyperparameters per model (Table VII), concluding that the pipeline efficiently identifies optimal configurations and generalizes beyond transformers. The code and full experimental results are released on GitHub.","tokens_in":9130,"tokens_out":4733,"duration_ms":47008,"significance":"If the empirical results are reliable, the paper provides practical guidance for practitioners and a reusable pipeline; strengths include open code and results, standard benchmark datasets, and coverage of six models including two non-transformer baselines. However, the significance is at present conditional: the central quantitative findings rest on only 20 single-seed trials per model-dataset pair, ad hoc exclusion of OOM-crashed and outlying trials, and visually inferred importance rankings, so the reported best models and hyperparameter guidance are not yet established with the confidence the paper claims.","major_comments":[{"comment":"The HPO budget is only 20 Optuna trials per model-dataset pair, with no seeds or repeated runs. Against the discrete search spaces in Table III (eight hyperparameters, several with ten or more values), 20 trials cannot reliably locate near-optimal configurations, and training stochasticity is uncontrolled. The best-MSE column in Table IV may therefore reflect search luck rather than model quality, so the cross-model rankings (e.g., TimeMixer best on ETTh1, Crossformer best on Weather/ECL) are not load-bearing as reported. Repeating each tuning run with multiple seeds and reporting mean and dispersion (or IQR) is necessary.","section":"Section III-B.2 and Table IV"},{"comment":"Excluding OOM-crashed trials biases the comparison. On ECL, TimeMixer crashes in 85% of trials, PatchTST in 40%, and Mamba on Weather in 53%, so the 'best' results are often selected from only a few successful configurations (3-15 trials). Since OOM is systematically tied to batch size, d_model, and d_ff (Table VI), the effective search space for high-OOM models is restricted to a memory-feasible subset, meaning the benchmark compares unequal effective search budgets. Please report all trials, including failures, or use a memory-aware search space, and show whether the surviving configurations are representative.","section":"Section IV-B and Tables V-VI"},{"comment":"For Autoformer on Weather, outliers are removed post hoc before deriving parameter importance: 'we remove outliers from Autoformer's results and observed that batch size is the most influential parameter'. No definition, count, or threshold for these outliers is given. This is not reproducible and can change the conclusion; it should be replaced by a pre-specified robust analysis (e.g., median-based ranking or an explicit outlier criterion applied uniformly to all models) or removed.","section":"Section IV-B.3"},{"comment":"The top-3 hyperparameter importance ranking in Table VII is obtained by visual inspection of parallel coordinate plots, not by a quantitative sensitivity measure. With only 20 trials per model-dataset pair (and fewer after OOM exclusions), visual patterns can be driven by a few points, so the rankings are assertions rather than established findings. Please compute a quantitative importance score (e.g., fANOVA, ablation around the best configuration, or permutation importance on a fitted surrogate) and report the number of trials behind each ranking.","section":"Section IV-B.2 and Table VII"}],"minor_comments":[{"comment":"There are several typos and formatting issues: '2rd' and '3th' in the author affiliations, 'Hyperparmater' in the Section IV-B heading, and the Section IV-B title should read 'Hyperparameter and Model Metric'.","section":"Front matter and Section IV-B"},{"comment":"Table II is extremely difficult to read: the column headers mix model names with dataset names, and many cells contain repeated 'DF' values without clear meaning. It should be completely reformatted, with model names as columns and parameters as rows, and 'DF' defined.","section":"Table II"},{"comment":"In Table III, the 'learningrate' row leaves the Lowest and Highest Value columns empty, and 'Searching Space' should be 'Search Space'. Also, the relationship between the listed lowest/highest values and the one-step extension described in Section III-C should be stated explicitly for each parameter.","section":"Table III"},{"comment":"The captions for Figures 3 and 4 are uninformative: Figure 3 repeats 'ETTh1, Weather, ECL' under every panel, and Figure 4 has only 'Parallel coordinates plot on Weather dataset: Autoformer without outlier'. Please give each panel a descriptive caption explaining what is plotted.","section":"Figures 3-6"},{"comment":"The GitHub URL in the footnote appears as '1https://github.com/jingjing-unilu/HPO transformer time series' with a space and no visible hyperlink formatting. Please provide a working, properly rendered URL.","section":"Abstract and footnote 1"},{"comment":"The claim that the pipeline 'is generalizable beyond transformer-based architectures' is supported only by two non-transformer models (Mamba and TimeMixer) at a single prediction length (96). Please soften the claim or add evidence at other horizons and for additional non-transformer architectures.","section":"Section V and abstract"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper's pipeline itself is straightforward (a wrapper around Ray Tune and Optuna), so the main contribution is empirical. With only 20 single-seed trials per model-dataset pair and high OOM exclusions, the headline benchmark results (Table IV) and hyperparameter rankings (Table VII) are not yet convincing. The paper has useful assets—public code, full results, and a clear pipeline description—and the central idea is defensible; however, the empirical demonstration needs substantial rework before it can support the claims. I would not reject outright, but the authors should be asked to re-run with multiple seeds, handle OOM/outlier exclusions transparently, and replace visual importance rankings with a quantitative analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straightforward engineering paper. Take Optuna, Ray Tune, and W&B, wire them into a pipeline, run 20 trials per model-dataset pair on six TSF models and three datasets, report best MSE and eyeball parallel-coordinate plots. The pipeline itself is fine; the evidence for the claimed insights is not.\n\nWhat the paper does well: it reports OOM failure rates honestly, which is genuinely useful practical information, and it is transparent about the search-space design. The parallel-coordinate plots are a reasonable way to generate hypotheses, and the paper does not oversell the pipeline as a new algorithm.\n\nThe soft spots are real and load-bearing. The central empirical claims—best model per dataset and the top-3 hyperparameter importance rankings in Table VII—rest on 20 Optuna trials per model-dataset pair, run once with no seed control. The exclusions are severe: TimeMixer on ECL loses 85% of trials, Mamba on Weather loses 53%, so the \"best\" result for those cells comes from as few as 3–10 successful trials. That makes the cross-model comparison in Table IV uneven: models with high OOM rates are effectively searching over a restricted memory-feasible subspace, not the same search space as others. The importance rankings come from visual inspection of parallel-coordinate plots, with no quantification, and 20 (or fewer) points over an 8-parameter space is far too sparse for that to be reliable. The Autoformer Weather analysis removes outliers post hoc without saying what was removed. And the code link in the footnote has spaces in the URL—\"HPO transformer time series\"—so it does not resolve, which undercuts the reproducibility promise.\n\nNone of this means the pipeline is broken. It probably works as described and would help practitioners get a starting point for tuning TSF models. But the paper's presentation overstates what the data supports. The rankings should be labeled as rough, dataset-specific observations, not generalizable guidance, and the cross-model \"winner\" claims need error bars or at least multiple seeds.\n\nA modest revision could fix most of this: repeat each configuration a few times, report variance, handle OOM systematically (e.g., as a penalty rather than exclusion), and replace visual rankings with a simple sensitivity measure. As it stands, this is a paper for practitioners who want a template for HPO on TSF models, not a paper with strong scientific conclusions. I would send it to a serious referee rather than desk reject it, because the OOM analysis and search-space design have practical value and the weaknesses are fixable. I would not cite it in my own work in the next year.","headline":"A workmanlike HPO pipeline paper assembled from existing tools; the pipeline is plausible, but the empirical insights rest on 20 single-seed trials with severe OOM exclusions, so the rankings and cross-model comparisons are not reliable as presented.","tokens_in":9607,"tokens_out":2252,"would_cite":false,"duration_ms":23398,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single hyperparameter-optimization pipeline, built on Optuna and Ray Tune, efficiently finds optimal configurations for transformer-based time series forecasting models and also works for non-transformer models…","keywords":["hyperparameter optimization","time series forecasting","transformer models","OptunaSearch","Ray Tune","benchmark","Mamba","TimeMixer"],"falsifier":"Re-run the pipeline on the same three datasets with 100 trials per model-dataset pair and several seeds per configuration, and compare the best models and top-three hyperparameter rankings; if TimeMixer no longer leads on ETTh1, Crossformer no longer leads on Weather and ECL, or the importance order changes, the reported conclusions were artifacts of the small sample and exclusions.","tokens_in":8650,"feed_emoji":"⚙️","tokens_out":7019,"duration_ms":62685,"temperature":0.7,"pith_summary":"This paper tries to establish that one reusable hyperparameter-optimization pipeline can efficiently find good hyperparameter settings for transformer-based time series forecasting models. The authors build the pipeline around a tree-structured Parzen estimator search, run 20 trials per model-dataset pair on ETTh1, Weather, and Electricity, and report which hyperparameters matter most. They show that the same pipeline tunes non-transformer state-of-the-art models, Mamba and TimeMixer, as well. If the claim holds, practitioners and researchers can follow the pipeline instead of hand-tuning each model and dataset.","feed_headline":"One tuning pipeline finds best settings for six forecasting models","feed_subtitle":"On ETTh1, Weather, and ECL, it tunes transformers plus Mamba and TimeMixer; key knobs: model size, learning rate, batch size.","key_machinery":"The load-bearing mechanism is the HPO pipeline itself: it starts from the code base's parameter set, selects common hyperparameters, builds a search space whose lower and upper bounds are one step beyond the minimum and maximum values seen across models, and searches with OptunaSearch, a tree-structured Parzen estimator variant, executed under Ray Tune. The same pipeline logs each trial, removes out-of-memory crashes, and uses parallel-coordinate plots to map hyperparameter values to validation loss, which generates the importance rankings. This is what carries the claim that a single workflow can replace per-model manual tuning.","core_discovery":"The central claim is that a unified pipeline—collecting common hyperparameters, defining a search space one step wider than the models' existing ranges, and searching with OptunaSearch under Ray Tune—identifies competitive configurations for four transformer forecasters (Autoformer, Crossformer, Non-Stationary Transformer, PatchTST) and two other state-of-the-art models (Mamba, TimeMixer) on ETTh1, Weather, and ECL. In the reported runs, TimeMixer has the lowest validation error on ETTh1 while Crossformer leads on Weather and ECL. The analysis concludes that model size ($d_{model}$), learning rate, and batch size are the most influential hyperparameters, and that high-dimensional datasets push many trials into out-of-memory failures, so future tuning should constrain model size and batch size or use distributed GPUs.","pith_inferences":["A fair test would rerun the same pipeline with many more trials and multiple seeds; the paper's 20 single-run trials leave the reported model rankings and parameter importance orders open to sampling noise.","The underfitting curves on ETTh1 and Weather suggest the pipeline's chosen epoch range and model-capacity bounds may cap achievable accuracy; extending the search space could change which model wins.","The same workflow could be paired with a cost axis—Crossformer's best Weather and ECL results come at much larger training times—to support multi-objective tuning that trades accuracy against compute.","Applying the pipeline to prediction lengths 192, 336, and 720 would test whether the importance rankings transfer across forecast horizons."],"forward_implications":["A practitioner can apply the same pipeline to a new transformer-based forecasting model or dataset and get a tuned configuration without designing a bespoke search.","Because the pipeline extends to Mamba and TimeMixer, its usefulness is not tied to a particular attention architecture.","On datasets with many variables, tuning runs that combine large batch sizes with large model widths will frequently crash, so memory-aware search spaces are a primary constraint.","The reported top parameters—$d_{model}$, learning rate, and batch size—give concrete starting points for subsequent tuning rounds."],"supporting_citations":[{"why":"Provides the open-source code base that the pipeline extends and executes.","marker":"[29]"},{"why":"Supplies Autoformer, one of the transformer models benchmarked under the pipeline.","marker":"[28]"},{"why":"Supplies Crossformer, the model reported best on Weather and ECL.","marker":"[34]"},{"why":"Supplies Non-Stationary Transformer, one of the benchmarked transformer models.","marker":"[14]"},{"why":"Supplies PatchTST, one of the benchmarked transformer models.","marker":"[16]"},{"why":"Supplies Mamba, a non-transformer model used to show the pipeline generalizes.","marker":"[9]"},{"why":"Supplies TimeMixer, a non-transformer model and best on ETTh1.","marker":"[23]"},{"why":"Introduces the ETTh1 dataset and benchmark conventions used in the experiments.","marker":"[35]"}],"fun_headline_variants":["Unified pipeline tunes six forecasters, pinpoints key hyperparameters","HPO for transformers, Mamba, TimeMixer: one search space to rule them","Model size, learning rate, batch size: what really matters in TSF tuning","From transformers to Mamba: a generalizable tuning recipe for TSF"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that twenty tuning attempts per model and dataset, each run once without seed control and with crashed or outlier trials removed, give a reliable picture of each model's best achievable performance and of which hyperparameters matter.","fun_headline_variants_meta":{"raw":{"variants":["Unified pipeline tunes six forecasters, pinpoints key hyperparameters","HPO for transformers, Mamba, TimeMixer: one search space to rule them","Model size, learning rate, batch size: what really matters in TSF tuning","From transformers to Mamba: a generalizable tuning recipe for TSF"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":3968,"prompt_tokens":887,"completion_tokens":3081,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2997}},"tokens_in":503,"tokens_out":3081,"duration_ms":21951,"temperature":1.0,"reasoning_tokens":2997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:28:03.668440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the same three datasets with 100 trials per model-dataset pair and several seeds per configuration, and compare the best models and top-three hyperparameter rankings; if TimeMixer no longer leads on ETTh1, Crossformer no longer leads on Weather and ECL, or the importance order changes, the reported conclusions were artifacts of the small sample and exclusions.","supporting_citations":[{"cited_title":"TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis","cited_arxiv_id":null,"evidence_quote":"Provides the open-source code base that the pipeline extends and executes."},{"cited_title":"Autoformer: decomposition transform- ers with auto-correlation for long-term series forecast- ing","cited_arxiv_id":null,"evidence_quote":"Supplies Autoformer, one of the transformer models benchmarked under the pipeline."},{"cited_title":"Crossformer: Trans- former utilizing cross-dimension dependency for multi- variate time series forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies Crossformer, the model reported best on Weather and ECL."},{"cited_title":"Non-stationary transformers: Exploring the stationarity in time series forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies Non-Stationary Transformer, one of the benchmarked transformer models."},{"cited_title":"A Time Series is Worth 64 Words: Long- term Forecasting with Transformers","cited_arxiv_id":null,"evidence_quote":"Supplies PatchTST, one of the benchmarked transformer models."},{"cited_title":"TimeMixer: Decomposable Multi- scale Mixing for Time Series Forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies TimeMixer, a non-transformer model and best on ETTh1."},{"cited_title":"Informer: Beyond efficient trans- former for long sequence time-series forecasting","cited_arxiv_id":null,"evidence_quote":"Introduces the ETTh1 dataset and benchmark conventions used in the experiments."}],"review_version":1}