{"id":"2a2cda47-0399-4dc1-9594-48b326ec8118","arxiv_id":"2501.07034","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Chronos, a pretrained time series foundation model, predicts car-following acceleration better than an uncalibrated IDM baseline and comparably to deep learning baselines on the Open ACC dataset.","lead":"Researchers applied Amazon's Chronos, a pretrained time series foundation model, to forecast the acceleration of a car following an adaptive cruise control (ACC) vehicle, using the Open ACC dataset. Off-the-shelf Chronos reached an RMSE of 0.60, matching deep learning baselines, while fine-tuning with covariates improved it to 0.53, about 34% better than the IDM baseline in this test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IDM and Chronos appear to be evaluated on different prediction tasks; the 33.75% improvement is not established without a shared backtesting protocol.","rationale":"After reading the full text, I find the reader's verdict of CONDITIONAL appropriate, but for a slightly different reason than the stated weakest assumption. The IDM calibration issue is real, but a more fundamental problem is that the evaluation protocols for IDM and Chronos are never demonstrated to be the same. The paper describes a sophisticated backtesting/forecasting setup for the time-series models but treats IDM as a simple simulator with no description of how it is embedded in that setup. If IDM is being asked to predict acceleration from the current state while Chronos is asked to forecast 3 seconds ahead from history, the comparison is unfair regardless of calibration. The abstract's quantitative claims therefore rest on an unverified comparability assumption. The paper does have independent value as an exploratory application: zero-shot Chronos at 0.60 RMSE is an interesting result, and the fine-tuning gain over zero-shot (0.60 to 0.53) is internally consistent. But the headline percentages against IDM and other baselines cannot be assessed from the present manuscript. A rerun under a shared protocol, plus a table of per-model RMSE values, would settle the issue. I therefore keep the verdict at CONDITIONAL/unchanged, but the condition should include evaluation-protocol consistency, not just IDM calibration.","tokens_in":11627,"tokens_out":6848,"duration_ms":64038,"concrete_test":"Rerun the full comparison under one explicitly shared protocol: for each test trajectory and each backtest window, provide every model with the same observed 6-second history and the same covariate information available at the forecast origin (e.g., current gap and speeds), and require IDM to forecast the same 3-second-ahead acceleration sequence from that origin without access to future states. Report per-model RMSE in a table. If the IDM RMSE under this protocol differs from 0.80, or if any deep-learning baseline RMSE changes, the reported 33.75% improvement and 12–37% range are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (abstract; Results) is that fine-tuned Chronos achieves RMSE 0.53, a 33.75% improvement over IDM and a 12–37% reduction over ETS/DeepAR/WaveNet/TFT. This claim requires all models to be evaluated on the same forecasting task. The paper does not show that. In the Evaluation Metric section, the time-series models are evaluated with a multi-window backtest: 6 seconds of context, 3-second forecast horizon, repeated over the test set. The IDM subsection, however, states only that the first 80% of trajectories are used for training and the remaining 20% for testing, then RMSE of predicted a_f is computed; no backtesting protocol is described for IDM. It is therefore unclear whether IDM predicts the same 3-second-ahead acceleration as Chronos, or instead computes an instantaneous/one-step acceleration from the currently observed gap and speeds. If the latter, the RMSE values are not comparable and the headline improvement is an artifact of task mismatch rather than model capability. The same ambiguity affects the deep-learning baselines, and the abstract's '12–37%' range is not backed by a per-model results table in the text. The paper also leaves unclear how covariates are made available to Chronos+LightGBM during forecasting, and whether the LightGBM residual model is fit on training-fold residuals only. These gaps make the central comparison unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes using Amazon's Chronos time-series foundation model to forecast the acceleration of the following vehicle in the Open ACC Casale car-following dataset. It compares zero-shot and fine-tuned Chronos (with a LightGBM residual model to incorporate covariates) against IDM, ETS, DeepAR, WaveNet, and TFT, reporting RMSE. The headline claim is that fine-tuned Chronos achieves RMSE 0.53, a 33.75% improvement over IDM and a 12–37% reduction over the other baselines.","tokens_in":11958,"tokens_out":4534,"duration_ms":40191,"significance":"If the comparison is made rigorous, this would be a useful first demonstration that pretrained time-series foundation models can be applied to car-following behavior without training from scratch, with zero-shot performance competitive with task-specific deep learning models and fine-tuning on a small dataset providing further gains. The study is clearly exploratory: it uses one dataset and one foundation model. The core quantitative claim is not yet fully supported, but the direction is worth publishing after substantial revision.","major_comments":[{"comment":"The IDM and time-series models are not shown to be evaluated on the same forecasting task. The Evaluation Metric section describes a multi-window backtest for the time-series models (6 seconds of context, 3-second forecast horizon, repeated over the test set; Eqs. 11-12). The Intelligent Drive Model section, in contrast, only states that the first 80% of trajectories are used for training and the remaining 20% for testing, with RMSE of the predicted acceleration computed. No backtesting protocol is described for IDM. If IDM predicts the current-step acceleration from the observed gap and speeds rather than the 3-second-ahead forecast, the RMSE values are not comparable and the headline 33.75% improvement would be an artifact of task mismatch. Please specify and apply a single shared evaluation protocol to all models.","section":"Evaluation Metric / Intelligent Drive Model"},{"comment":"The IDM baseline parameters are taken from the literature, not calibrated to the Open ACC training split. The text states that parameter values and ranges come from Treiber, Milanés, Kim, and Souza, and no calibration procedure is described. Because the paper's central claim is an improvement over IDM, the comparison is only meaningful if IDM is fitted to the training data or the authors justify why the literature values are a strong baseline. Please calibrate IDM parameters on the training split (or report both zero-shot and calibrated IDM RMSE) and document the calibrated values.","section":"Intelligent Drive Model"},{"comment":"The reported performance of the deep learning baselines is not backed by per-model results. Figure 2 is described in the text, but the text only gives RMSE values for IDM (0.80), ETS (0.84), zero-shot Chronos (0.60), and fine-tuned Chronos (0.53). The abstract's claim of a 12–37% reduction over DeepAR, WaveNet, and TFT requires a table listing each model's mean RMSE and standard deviation. In addition, no statistical significance test accompanies the 0.60 vs 0.53 difference, so it is unclear whether the fine-tuning gain is meaningful. Provide a results table with per-model RMSE across backtest windows and a significance test or confidence intervals.","section":"Results / Figure 2"},{"comment":"The manuscript does not actually display the equations it references: Equations 1–12 appear as blank placeholders in the text. Without the IDM formula, the covariate integration scheme (Eqs. 4-10), and the backtesting aggregation (Eqs. 11-12), the methods are not reproducible. The experimental configuration is also incomplete: the context length and forecast horizon are stated as 6 and 3 seconds in the description of Figure 1, but the Chronos model size used for the headline result, the fine-tuning procedure (epochs, learning rate, data split by trajectory), and LightGBM hyperparameters are not given. Please include all equations and a complete experimental configuration.","section":"Methods (Eqs. 1-12)"}],"minor_comments":[{"comment":"The text refers to the statistical baseline as 'EST' in the Results section; this should be 'ETS'.","section":"Results"},{"comment":"The caption says 'greed boxes' where it should say 'green boxes'.","section":"Figure 1 caption"},{"comment":"The phrase 'out-of-the-pocket capability' should be 'out-of-the-box capability'.","section":"Introduction"},{"comment":"Reference numbering is inconsistent: 'Hong et al. (33)' and 'Darlow et al. (35)' point to references 33 and 35, but those entries are Kim and Heaslip and Moor et al., respectively; the works by Hong et al. and Darlow et al. are not in the reference list.","section":"Related Work / References"},{"comment":"Table 2 has blank placeholders for variable names and units, making the table unreadable; please fill in the missing entries.","section":"Table 2"},{"comment":"The data-cleaning description states that acceleration and Frenet position are calculated, but the exact cleaning steps and formulas from Zhou et al. are not described; please specify the processing procedure.","section":"Data Overview and Process"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to require substantial revision before meeting the standard of a serious archival journal. The strengths are the novelty of applying a time-series foundation model in this domain and the use of an open dataset; the main weaknesses are the lack of a rigorous, shared evaluation protocol and the many presentation gaps (missing equations, empty table placeholders). The manuscript appears to be a preliminary exploratory study; a workshop or a transportation-domain venue emphasizing applications may be a better fit than a general ML venue in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this is a proof-of-concept paper: it applies Amazon's Chronos time series foundation model to acceleration forecasting of a following vehicle in the Open ACC dataset, and it appears to be the first to do so. That alone is worth something. The authors compare zero-shot and fine-tuned Chronos against IDM, ETS, DeepAR, WaveNet, and TFT, and they add a LightGBM residual correction to incorporate covariates like gap and speed difference. The results are plausible: zero-shot Chronos at RMSE 0.60, fine-tuned at 0.53, with IDM at 0.80 and ETS at 0.84. The paper is honest about being a single-dataset exploration, and the writing is straightforward.\n\nThe soft spots are real, and they cluster around the comparison protocol. The IDM baseline uses parameter values from the literature (Treiber, Milanés, Kim, Souza) and is not fitted on the training split. That alone would explain part of the 33.75% gap. More concerning, the paper never explicitly states that IDM is evaluated under the same multi-window backtesting used for the time series models. The IDM section only says 80% training / 20% testing and then RMSE; it may be predicting instantaneous acceleration from current state rather than 3-second-ahead acceleration. If so, the headline comparison is apples-to-oranges. The same ambiguity applies to the deep learning baselines. The '12-37% reduction' is not backed by a per-model table; only IDM and ETS numbers appear in the text.\n\nThe remaining issues are smaller but still matter for a benchmark paper: no significance tests, so the 0.60 vs. 0.53 difference could be noise; no code or data release; a few typos (ETS spelled EST once). None of these are fatal. The core idea — a pretrained foundation model plus a light residual model can do reasonable car-following forecasting with little data — is sound and worth testing properly. The paper deserves a serious referee, but it needs major revision before the empirical claims can be trusted: calibrate IDM on the training split, document the forecasting task for every baseline, report per-model RMSEs, add error bars or significance tests, and release code.\n\nIf you work on traffic simulation or ACC behavior analysis, this is a useful pointer; otherwise it's a decent example of how not to compare a new model against a weak baseline. I'd bring it to a reading group for the methodology discussion.","headline":"First benchmark of Chronos on car-following data, but the headline 33.75% improvement over IDM is not yet supported because the IDM baseline is uncalibrated and the evaluation tasks may not be aligned.","tokens_in":12431,"tokens_out":2913,"would_cite":false,"duration_ms":27234,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pretrained time series foundation model, fine-tuned on the Open ACC dataset, forecasts car-following acceleration with an RMSE of 0.53, a 33.75% improvement over the Intelligent Driver Model.","keywords":["car-following","time series foundation models","Chronos","Open ACC dataset","acceleration prediction","Intelligent Driver Model","fine-tuning","traffic simulation"],"falsifier":"Calibrate IDM's parameters on the training split (instead of taking values from prior literature) and measure RMSE on the same test trajectories; if the fitted IDM reaches or beats 0.53, the headline improvement over IDM collapses.","tokens_in":11438,"feed_emoji":"🚗","tokens_out":9278,"duration_ms":80446,"temperature":0.7,"pith_summary":"The paper tries to show that a general-purpose pretrained time series model, Chronos, can be used for car-following behavior prediction without being trained from scratch on traffic data. On the Open ACC dataset's Casale experiment, zero-shot Chronos forecasts follower acceleration with a mean RMSE of 0.60, on par with deep learning baselines like DeepAR, WaveNet, and TFT. Fine-tuning Chronos on 80% of the trajectories and adding covariate corrections drops the RMSE to 0.53, a 33.75% improvement over an IDM baseline and a 12-37% reduction relative to the other baselines. The authors read this as evidence that foundation models offer a scalable, adaptable alternative for transportation trajectory forecasting, and they present it as the first application of time series foundation models to car-following behavior.","feed_headline":"Pretrained model cuts car-following forecast error by 33.75%","feed_subtitle":"Zero-shot Chronos matches deep learning baselines; fine-tuned with covariates it reaches RMSE 0.53 on the Open ACC dataset.","key_machinery":"The engine of the paper is Chronos, a pretrained probabilistic time series model that scales and quantizes time series values into a fixed token vocabulary and trains a text-to-text transformer-style language model with cross-entropy loss; forecasts are sampled autoregressively. Around this, the paper builds a residual-correction mechanism: Chronos produces a base forecast of follower acceleration, then a gradient-boosted tree model (LightGBM) fits the forecast residuals using dynamic covariates (space gap, speed difference, follower speed). Multi-window backtesting, rolling a six-second history forward in three-second steps, supplies the RMSE numbers that grade every model.","core_discovery":"On the Casale experiment of the Open ACC dataset, Chronos without any fine-tuning predicts following-vehicle acceleration with mean RMSE around 0.60, matching the accuracy of task-specific deep learning models (DeepAR, WaveNet, TFT) and beating the traditional IDM and ETS baselines. With fine-tuning on the training split, Chronos reaches a mean RMSE of 0.53, which the paper reports as a 33.75% improvement over IDM (0.80) and a 12-37% reduction over ETS and the deep learning baselines. The best configuration adds dynamic covariates (space gap, speed difference, follower speed) through a residual-correction step, and the smallest 46-million-parameter Chronos performs as well as the 200-million and 710-million versions. The paper claims this demonstrates that pretrained time series foundation models can be adapted to car-following analysis with minimal data and without the parameter-calibration burden of traditional car-following models.","pith_inferences":["A natural test the paper leaves for future work is cross-dataset evaluation: fine-tune on one ACC dataset and evaluate on a held-out one; a persistent gain would support general driving dynamics rather than dataset-specific fitting.","The residual-correction wrapper is model-agnostic; applying it to other time series foundation models would clarify whether the gain comes from Chronos or from the covariate adapter.","Since the smallest model already saturates performance, larger-scale versions may not help this short-horizon task; the limit may be the information content of the inputs rather than model capacity."],"forward_implications":["Fine-tuning a pretrained time series foundation model on a modest trajectory dataset can replace training a deep learning car-following model from scratch while improving accuracy.","The 46-million-parameter Chronos variant is sufficient for this task, so deployment on modest hardware is plausible.","Because the forecast is produced from historical acceleration plus covariate corrections, it no longer requires knowing the leader's current speed at inference, which traditional car-following formulas require.","The same recipe (pretrained forecaster plus residual correction on covariates) is a template for other transportation time series tasks like traffic flow or speed prediction."],"supporting_citations":[{"why":"Defines the Intelligent Driver Model used as the main traditional baseline for acceleration prediction.","marker":"Treiber et al. (4)"},{"why":"Supplies IDM parameter values and ranges used to configure the baseline.","marker":"Milanés et al. (7)"},{"why":"Another source of IDM parameters for ACC vehicle car-following, used to set the baseline.","marker":"Souza et al. (38)"},{"why":"Introduces Chronos, the time series foundation model that the paper fine-tunes and evaluates.","marker":"Ansari et al. (18)"},{"why":"Provides the Open ACC dataset, source of the Casale car-following trajectories used in all experiments.","marker":"Anesiadou et al. (2020)"},{"why":"Provides the data cleaning and processing steps the paper follows to build trajectory records.","marker":"Zhou et al. (37)"},{"why":"Motivates the choice of space gap, speed difference, and follower speed as input covariates for car-following prediction.","marker":"Ma et al. (9)"},{"why":"Defines DeepAR, one of the deep learning baselines that zero-shot Chronos matches and fine-tuned Chronos outperforms.","marker":"Salinas et al. (42)"},{"why":"Defines WaveNet, another deep learning baseline used for comparison.","marker":"Oord et al. (43)"},{"why":"Defines the Temporal Fusion Transformer baseline used for comparison.","marker":"Lim et al. (44)"}],"fun_headline_variants":["Zero-shot Chronos matches deep nets on car-following","Chronos foundation model cuts car-following error by 33.75%","Foundation model rivals deep learning for car-following without tuning","Time series model beats IDM on car-following after minimal tuning","Pretrained Chronos: car-following error drops to 0.53 RMSE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed improvement over IDM assumes the IDM baseline is strong, but the paper does not report calibrating IDM parameters on the training split, so IDM might underperform a properly fitted version and inflate the reported gain.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot Chronos matches deep nets on car-following","Chronos foundation model cuts car-following error by 33.75%","Foundation model rivals deep learning for car-following without tuning","Time series model beats IDM on car-following after minimal tuning","Pretrained Chronos: car-following error drops to 0.53 RMSE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3600,"prompt_tokens":1047,"completion_tokens":2553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":663,"tokens_out":2553,"duration_ms":19392,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:06.528193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Calibrate IDM's parameters on the training split (instead of taking values from prior literature) and measure RMSE on the same test trajectories; if the fitted IDM reaches or beats 0.53, the headline improvement over IDM collapses.","supporting_citations":[],"review_version":1}