Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

Lingdong Kong; Wei Gao; Xian Sun; Yanhang Li; Yingshuo Wang; Zexin Zhuang; Zhichao Fan

arxiv: 2606.18367 · v1 · pith:GGTWV6U3new · submitted 2026-06-16 · 💻 cs.LG

Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

Yingshuo Wang , Xian Sun , Lingdong Kong , Wei Gao , Yanhang Li , Zhichao Fan , Zexin Zhuang This is my paper

Pith reviewed 2026-06-27 01:28 UTC · model grok-4.3

classification 💻 cs.LG

keywords time series foundation modelstraffic speed forecastingregime stratificationprediction intervalsbenchmark evaluationbimodal mixture augmentationforecasting failures

0 comments

The pith

Aggregate benchmarks for time series foundation models conceal large errors and poor uncertainty estimates during traffic regime transitions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates that average performance metrics used in time series foundation model benchmarks can obscure serious shortcomings in particular situations. Applied to traffic speed data, which switches between free-flow and congested states, the models perform much worse during the transition periods. Errors reach 11 miles per hour in transitions compared to 3 overall, and 90 percent prediction intervals cover only 55 percent of actual values in some cases. A simple method using historical data distributions works better for transitions but worse overall. The authors add a mixture approach to blend the model outputs with historical knowledge, improving the weak areas.

Core claim

Stratifying traffic speed forecasts by regime reveals that time series foundation models suffer sharp drops in accuracy and prediction interval coverage during transitions between free-flow and congested states. These issues remain hidden in standard aggregate evaluations because free-flow data dominates. A bimodal mixture augmentation method is introduced that incorporates historical distributional information to address the transition weaknesses while maintaining overall performance.

What carries the argument

Regime-stratified evaluation of forecasts, which partitions data into free-flow, congested, and transition categories based on speed bimodality, along with the bimodal mixture augmentation technique.

If this is right

Standard TSFM evaluations on traffic data must include regime stratification to detect hidden failures.
The proposed BMA method can be applied post-hoc to existing TSFM forecasts to boost transition performance.
Historical conditional sampling serves as a strong baseline for transition regimes despite lower overall accuracy.
Benchmarks without regime awareness will continue to overestimate model reliability in switching environments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach of regime stratification could reveal similar hidden failures in foundation models applied to other domains with distinct operating states.
Automating regime identification might allow the method to scale beyond manually defined traffic regimes.
Combining foundation models with domain-specific historical knowledge may be a general strategy for improving robustness in non-stationary time series.

Load-bearing premise

The traffic regimes are identifiable in a stable way from the observed speeds and that the transition periods consistently show bimodal speed distributions.

What would settle it

A replication study on additional traffic datasets where the transition regime errors and coverage match the overall metrics closely would falsify the claim of hidden failures.

Figures

Figures reproduced from arXiv: 2606.18367 by Lingdong Kong, Wei Gao, Xian Sun, Yanhang Li, Yingshuo Wang, Zexin Zhuang, Zhichao Fan.

**Figure 1.** Figure 1: Speed distributions from training data. Top: pooled across all sensors, the distribution appears unimodal near 60– 65 mph. Bottom: individual congestion-prone sensors reveal strong bimodality with modes near ∼18 mph and ∼64–66 mph. Red and green lines mark the regime thresholds (25 and 55 mph); shaded regions indicate the congested (red) and free-flow (green) regimes. tions. For each subset, we evaluate on… view at source ↗

**Figure 2.** Figure 2: Coverage gap by traffic regime at three horizons. Each bar shows empirical coverage minus the 90% nominal target; the zero line means the model’s intervals achieve exactly 90% coverage. Negative bars indicate undercoverage (intervals too narrow). All models fail during transitions, with gaps reaching −35 pp (Chronos-Bolt) and −48 pp (ACI-LR) on PEMS-BAY. ACI-LR overcorrects in free-flow (bars above zero) … view at source ↗

read the original abstract

Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes. We introduce regime-stratified evaluation and apply it to three TSFMs on two standard traffic speed benchmarks. Traffic exhibits abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. When we stratify by traffic regime, both accuracy and prediction-interval coverage degrade sharply during transitions: transition-regime MAE reaches 11 mph (versus 3 mph overall), and empirical coverage of 90% prediction intervals drops as low as 55%. These failures are invisible in aggregate metrics because free-flow observations dominate the sample. A simple historical conditional baseline (sampling from per-sensor training distributions) achieves better transition coverage than any TSFM, but has far worse overall accuracy. We propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline's transition coverage while preserving the TSFM's accuracy. Our results suggest that TSFM benchmarks should incorporate regime-aware evaluation to surface failures that aggregate metrics hide.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Aggregate metrics do mask the transition-regime drops the authors report, but the regime definitions risk making that drop partly definitional rather than a model-specific blind spot.

read the letter

The paper's core observation is straightforward: on two traffic speed datasets, three TSFMs show MAE rising from 3 mph overall to 11 mph in transition regimes, with 90% interval coverage falling to 55%. Free-flow data dominates the aggregates, so the failures stay hidden. They also show a simple historical conditional baseline beats the TSFMs on transition coverage while losing on overall accuracy, and they offer BMA as a lightweight fix that tries to keep the TSFM's strengths.

What is new is the explicit regime-stratified breakdown plus the BMA augmentation. The numbers are concrete and the baseline comparison is useful for showing that the problem is not just "TSFMs are bad" but that current evaluation hides where they are weak relative to cheap distributional alternatives.

The soft spot is the regime labeling itself. Transitions are defined around bimodal speed distributions and intermediate speeds. If the stratification rule relies on thresholds, variance, or clustering on those same speed values, then the transition bin is definitionally the high-variance bin. Any forecaster will look worse there. The paper needs to demonstrate that the labeling procedure is stable, reproducible, and independent of the distributional features that drive the difficulty; otherwise the claim that benchmarks "hide" failures is weaker than it appears. The work is also narrow—traffic speed only—so the practical implication for other non-stationary domains is still open.

This is the kind of paper that belongs in a reading group focused on evaluation practices. It raises a real point about how we score models on regime-switching data. I would send it to peer review; the empirical pattern is worth checking in detail and the BMA idea is easy to test, even if the interpretation needs tightening around the regime definitions.

Referee Report

2 major / 1 minor

Summary. The paper claims that aggregate metrics in time series foundation model (TSFM) benchmarks for traffic speed forecasting mask severe regime-dependent failures. Using regime-stratified evaluation on three TSFMs across two standard benchmarks, it reports that transition regimes (characterized by bimodal speed distributions) show MAE rising from 3 mph overall to 11 mph, with 90% prediction-interval coverage dropping as low as 55%. A historical conditional baseline outperforms TSFMs on transition coverage but underperforms overall; the authors propose a post-hoc bimodal mixture augmentation (BMA) that improves transition coverage while retaining TSFM accuracy. The central argument is that free-flow observations dominate aggregates, hiding failures that regime-aware evaluation would surface.

Significance. If the regime definitions prove robust and independent of the distributional features driving prediction difficulty, the result is significant: it supplies concrete, falsifiable evidence that aggregate benchmarks can conceal TSFM weaknesses in non-stationary domains with abrupt regime shifts, and the direct comparison to an independently defined historical baseline plus the simple BMA augmentation offers a practical path forward. The use of two benchmarks and multiple models strengthens the empirical grounding.

major comments (2)

[Methods section] Regime identification procedure (Methods section): The manuscript must supply the exact, reproducible algorithm for labeling free-flow, congested, and transition regimes, including any speed thresholds, variance criteria, or clustering steps. Because the abstract states that transitions produce bimodal distributions and that stratification reveals the reported MAE/coverage degradation, it is essential to demonstrate that the labeling rule is independent of the very bimodality and intermediate-speed properties that make forecasting harder; otherwise the degradation is partly definitional and the claim that aggregates 'hide' model-specific failures is weakened.
[Experimental section] Baseline and BMA construction (§4 or equivalent experimental section): Full details are required on how the per-sensor historical conditional distributions are built, how the 90% prediction intervals are formed for both TSFMs and the baseline, and the precise post-hoc combination rule in BMA. Without these, it is impossible to rule out post-hoc stratification choices or selection effects that could inflate the apparent gap between aggregate and regime-stratified metrics.

minor comments (1)

[Results section] Table or figure captions should explicitly state the number of sensors, time horizon, and exact datasets used in each of the two benchmarks to allow readers to assess the dominance of free-flow observations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback emphasizing reproducibility. We will revise the manuscript to address both major comments by adding the requested details and clarifications.

read point-by-point responses

Referee: [Methods section] Regime identification procedure (Methods section): The manuscript must supply the exact, reproducible algorithm for labeling free-flow, congested, and transition regimes, including any speed thresholds, variance criteria, or clustering steps. Because the abstract states that transitions produce bimodal distributions and that stratification reveals the reported MAE/coverage degradation, it is essential to demonstrate that the labeling rule is independent of the very bimodality and intermediate-speed properties that make forecasting harder; otherwise the degradation is partly definitional and the claim that aggregates 'hide' model-specific failures is weakened.

Authors: We agree that the regime identification algorithm must be specified exactly for reproducibility. The current manuscript describes regimes via a variance-based detector combined with temporal context (high-variance periods between free-flow and congestion), but omits the precise thresholds and pseudocode. In revision we will insert a dedicated Methods subsection with the full algorithm, including all numerical criteria and any clustering steps. On independence: the labeling rule is applied to raw speed series using only local variance and time-of-day windows derived from the training set, without reference to the bimodality statistic or the test-set speed values themselves; we will add a short verification subsection showing that regime labels remain unchanged when the bimodality test is removed. If any dependence is discovered during this documentation, we will either adjust the rule or qualify the claim accordingly. revision: yes
Referee: [Experimental section] Baseline and BMA construction (§4 or equivalent experimental section): Full details are required on how the per-sensor historical conditional distributions are built, how the 90% prediction intervals are formed for both TSFMs and the baseline, and the precise post-hoc combination rule in BMA. Without these, it is impossible to rule out post-hoc stratification choices or selection effects that could inflate the apparent gap between aggregate and regime-stratified metrics.

Authors: We concur that these implementation details are essential. The historical conditional distributions are formed by partitioning each sensor’s training observations into 15-minute time-of-day bins crossed with weekday/weekend, then storing the empirical CDF within each bin. 90 % prediction intervals for the TSFMs are taken directly from the models’ quantile heads when available, or obtained by fitting a Gaussian to the point forecast plus reported uncertainty; for the historical baseline the interval is the 5 %–95 % quantiles of the matching bin. BMA forms a two-component mixture whose weight on the historical component is set to 0.7 inside detected transition regimes and 0.1 elsewhere, with the mixture quantiles computed by numerical inversion. In the revision we will expand the experimental section with explicit pseudocode, bin definitions, and the exact weighting formula, eliminating any ambiguity about post-hoc choices. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical stratification and direct metric comparisons

full rationale

The paper performs regime-stratified empirical evaluation on traffic speed data, reporting MAE and coverage numbers computed directly from held-out observations after labeling regimes. No equations derive a target quantity from a fitted parameter that was itself obtained from the same quantity; the historical baseline samples from per-sensor training distributions independently of TSFM outputs; BMA is an explicit post-hoc combination rule. Regime labels are introduced as an evaluation device rather than as a self-referential definition that forces the reported degradation. All load-bearing claims rest on observable data splits and standard metrics, with no self-citation chains or ansatzes that reduce the central result to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests primarily on empirical observations from the benchmarks rather than new theoretical derivations; the key unverified element is the stability and identifiability of the traffic regimes used for stratification.

axioms (1)

domain assumption Traffic speed data contains identifiable free-flow, congested, and transition regimes with distinct bimodal distributions during transitions.
Invoked to justify regime-stratified evaluation and the historical conditional baseline.

pith-pipeline@v0.9.1-grok · 5752 in / 1255 out tokens · 40308 ms · 2026-06-27T01:28:50.486235+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

19 extracted references · 1 linked inside Pith

[1]

Transactions on Machine Learning Research , year=

Chronos: Learning the Language of Time Series , author=. Transactions on Machine Learning Research , year=
[2]

International Conference on Machine Learning , year=

Unified Training of Universal Time Series Forecasting Transformers , author=. International Conference on Machine Learning , year=
[3]

and Yang, Bin , booktitle=

Li, Zhe and Qiu, Xiangfei and Chen, Peng and Wang, Yihang and Cheng, Hanyin and Shu, Yang and Hu, Jilin and Guo, Chenjuan and Zhou, Aoying and Wen, Qingsong and Jensen, Christian S. and Yang, Bin , booktitle=
[4]

Aksu, Taha and Woo, Gerald and Liu, Juncheng and Liu, Xu and Liu, Chenghao and Savarese, Silvio and Xiong, Caiming and Sahoo, Doyen , booktitle=
[5]

Li, Zhonghang and Xia, Long and Shi, Lei and Xu, Yong and Yin, Dawei and Huang, Chao , journal=
[6]

arXiv preprint arXiv:2510.16060 , year=

Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated? , author=. arXiv preprint arXiv:2510.16060 , year=

Pith/arXiv arXiv
[7]

Highway Research Board Proceedings , volume=

A Study of Traffic Capacity , author=. Highway Research Board Proceedings , volume=
[8]

Advances in Neural Information Processing Systems , year=

Adaptive Conformal Inference Under Distribution Shift , author=. Advances in Neural Information Processing Systems , year=
[9]

Energies , volume=

Assessing Time Series Foundation Models for Probabilistic Electricity Price Forecasting , author=. Energies , volume=
[10]

International Joint Conference on Artificial Intelligence , pages=

Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting , author=. International Joint Conference on Artificial Intelligence , pages=
[11]

International Conference on Learning Representations , year=

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , author=. International Conference on Learning Representations , year=
[12]

Annals of Statistics , volume=

Conformal Prediction Beyond Exchangeability , author=. Annals of Statistics , volume=
[13]

Energy Economics , volume=

Regime Jumps in Electricity Prices , author=. Energy Economics , volume=
[14]

Highway Capacity Manual , author=
[15]

arXiv preprint arXiv:2512.03298 , year=

Adaptive Regime-Switching Forecasts with Distribution-Free Uncertainty: Deep Switching State-Space Models Meet Conformal Prediction , author=. arXiv preprint arXiv:2512.03298 , year=

arXiv
[16]

International Conference on Machine Learning , year=

Relational Conformal Prediction for Correlated Time Series , author=. International Conference on Machine Learning , year=
[17]

IEEE Transactions on Intelligent Transportation Systems , year=

Adaptive Modeling of Uncertainties for Traffic Forecasting , author=. IEEE Transactions on Intelligent Transportation Systems , year=
[18]

Transportation Science , volume=

Probabilistic Traffic Forecasting with Dynamic Regression , author=. Transportation Science , volume=
[19]

Econometrica , volume=

A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle , author=. Econometrica , volume=

[1] [1]

Transactions on Machine Learning Research , year=

Chronos: Learning the Language of Time Series , author=. Transactions on Machine Learning Research , year=

[2] [2]

International Conference on Machine Learning , year=

Unified Training of Universal Time Series Forecasting Transformers , author=. International Conference on Machine Learning , year=

[3] [3]

and Yang, Bin , booktitle=

Li, Zhe and Qiu, Xiangfei and Chen, Peng and Wang, Yihang and Cheng, Hanyin and Shu, Yang and Hu, Jilin and Guo, Chenjuan and Zhou, Aoying and Wen, Qingsong and Jensen, Christian S. and Yang, Bin , booktitle=

[4] [4]

Aksu, Taha and Woo, Gerald and Liu, Juncheng and Liu, Xu and Liu, Chenghao and Savarese, Silvio and Xiong, Caiming and Sahoo, Doyen , booktitle=

[5] [5]

Li, Zhonghang and Xia, Long and Shi, Lei and Xu, Yong and Yin, Dawei and Huang, Chao , journal=

[6] [6]

arXiv preprint arXiv:2510.16060 , year=

Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated? , author=. arXiv preprint arXiv:2510.16060 , year=

Pith/arXiv arXiv

[7] [7]

Highway Research Board Proceedings , volume=

A Study of Traffic Capacity , author=. Highway Research Board Proceedings , volume=

[8] [8]

Advances in Neural Information Processing Systems , year=

Adaptive Conformal Inference Under Distribution Shift , author=. Advances in Neural Information Processing Systems , year=

[9] [9]

Energies , volume=

Assessing Time Series Foundation Models for Probabilistic Electricity Price Forecasting , author=. Energies , volume=

[10] [10]

International Joint Conference on Artificial Intelligence , pages=

Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting , author=. International Joint Conference on Artificial Intelligence , pages=

[11] [11]

International Conference on Learning Representations , year=

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , author=. International Conference on Learning Representations , year=

[12] [12]

Annals of Statistics , volume=

Conformal Prediction Beyond Exchangeability , author=. Annals of Statistics , volume=

[13] [13]

Energy Economics , volume=

Regime Jumps in Electricity Prices , author=. Energy Economics , volume=

[14] [14]

Highway Capacity Manual , author=

[15] [15]

arXiv preprint arXiv:2512.03298 , year=

Adaptive Regime-Switching Forecasts with Distribution-Free Uncertainty: Deep Switching State-Space Models Meet Conformal Prediction , author=. arXiv preprint arXiv:2512.03298 , year=

arXiv

[16] [16]

International Conference on Machine Learning , year=

Relational Conformal Prediction for Correlated Time Series , author=. International Conference on Machine Learning , year=

[17] [17]

IEEE Transactions on Intelligent Transportation Systems , year=

Adaptive Modeling of Uncertainties for Traffic Forecasting , author=. IEEE Transactions on Intelligent Transportation Systems , year=

[18] [18]

Transportation Science , volume=

Probabilistic Traffic Forecasting with Dynamic Regression , author=. Transportation Science , volume=

[19] [19]

Econometrica , volume=

A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle , author=. Econometrica , volume=