{"id":"06c3525f-a386-4d36-bf81-ebbffdf21ddb","arxiv_id":"2607.04919","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Time-series foundation models are unconditionally better than classical methods on 15/30 datasets, lose early on 6, and a n_train<700 + seasonality rule resolves 10 deployment decisions without training.","lead":"Across 30 time-series datasets, foundation models beat classical methods at every data size on half the cases, while classical methods win with as little as 2% of data on others. A simple length-and-seasonality rule plus a small pilot tells practitioners when GPU foundation models are worth deploying versus XGBoost.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Best-of-FM vs best-of-classical selection, plus seed-sensitive sub-0.3 MASE gaps, overstates how often a real deployer can treat FM zero-shot as unconditionally dominant.","rationale":"The reader correctly isolates the post-hoc thresholds, discrete grid, best-of selection, and seed-sensitive margins as the soft spot; the paper’s own footnotes and Finding 1 already surface the same issues. That is the single most load-bearing concern for the strongest claim (15/30 unconditional FM wins + robust no-training rule resolving 10/30). No internal contradiction of the reported curves is evident, code is released, and the empirical patterns remain useful decision support once selection protocol is made explicit. Therefore the verdict stays CONDITIONAL with no upgrade or downgrade; the concrete recompute above is the minimal check that would settle whether the marketed counts survive a practitioner-realistic protocol. Agreement with the reader is full on the weakest assumption.","tokens_in":18331,"tokens_out":719,"duration_ms":6492,"concrete_test":"Recompute Table II and the 15/30 FM-dominant count under three fixed protocols: (i) mean MASE over the three seeds instead of min; (ii) each FM family alone (Chronos-Bolt only, Moirai only, Lag-Llama only) vs best classical; (iii) require gap ≥0.3 MASE (or bootstrap CI excluding zero) to keep FM-dominant. If the FM-dominant count falls below ~10/30 or the n_train<700 & S≥0.05 rule covers fewer than ~6 datasets without pilot, the unconditional-deployment and no-training-rule claims weaken and should be restated as selection-dependent.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central plurality claim (15/30 FM-dominant; GPU justified regardless of data volume) and the marketed n_train<700 & S≥0.05 rule rest on comparing the best zero-shot FM among Chronos/Moirai/Lag-Llama variants to the best classical method, using min-over-3-seeds MASE for FMs (Table II; §IV-A). The paper itself notes that a practitioner committed to a single FM family will typically see higher n* (Finding 1, §IV-B) and flags nine FM-dominant datasets with gaps <0.3 MASE that “may be reversed with additional seeds” (Table II footnote: Dominick Δ=0.008, ETTm1 0.066, Sunspot 0.087, etc.). Break-even is also defined only on a discrete 2/5/10/20/50/100% grid with FM context always full history while classical is fraction-restricted (§III-E). Thus the count of “unconditionally justified” deployments and the coverage of the no-training rule (10/30) are inflated relative to a fixed-model, mean-seed, continuous-n* decision a practitioner would actually make. The post-hoc 700/0.05 thresholds are acknowledged as cluster-derived (§IV-D), so the load-bearing risk is that the headline decision support over-claims robustness beyond this selection protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces break-even analysis for deciding when pretrained time series foundation models (Chronos, Moirai, Lag-Llama; zero-shot and LoRA) justify GPU deployment versus classical baselines (Naive, ETS, ARIMA, XGBoost). Across 30 datasets and six training fractions (2%–100%), it reports that zero-shot FMs beat the best classical method at every fraction on 15/30 datasets (FM-dominant), classical methods win with as little as 2% data on 6 datasets, and mid/late break-even occurs on the rest. It proposes a two-step rule: if n_train < 700 and seasonal strength S ≥ 0.05, use FM zero-shot and skip fine-tuning (resolving 10/30 cases without training); otherwise run a 5–10% pilot. LoRA is shown to degrade short series. Four features motivate exploratory hypotheses; LOO classifiers reach only 53–60% accuracy. Code and tools are released.","tokens_in":18740,"tokens_out":1244,"duration_ms":9410,"significance":"If the empirical map holds under realistic fixed-model deployment, the work supplies the first systematic, data-volume-aware answer to a high-stakes practitioner question that aggregate full-data FM benchmarks leave open. Strengths include a large transparent grid (~10,800 configurations), fixed-denominator MASE for cross-fraction comparability, chronological splits, public baselines, explicit caveats on seed-sensitive gaps and classifier limits, and released code/decision tools. The n_train/S rule and the finding that LoRA can hurt short series are immediately actionable even if the four-way taxonomy is partly post-hoc. The contribution is primarily empirical and decision-support oriented rather than theoretical.","major_comments":[{"comment":"Table II and §IV-A/B: the headline 15/30 FM-dominant count and “GPU unconditionally justified” claim rest on best-of-FM-family (min over 3 seeds) vs best-of-classical. Finding 1 itself notes that a practitioner committed to one FM family will typically see higher n*. Nine FM-dominant gaps are <0.3 MASE and flagged as seed-reversible (Dominick Δ=0.008, ETTm1 0.066, etc.). Report primary counts under fixed-model and mean-seed protocols, or relegate the plurality claim to a secondary best-of analysis, so the deployment rule is not overstated relative to real use.","section":null},{"comment":"§III-E and Eq. (1): break-even n* is observed only on a discrete {2,5,10,20,50,100}% grid, with FMs always given full historical context while classical methods are restricted to the current fraction. This systematically advantages FMs at low fractions and makes Early/Late labels grid-dependent. Sensitivity to denser grids, continuous interpolation, or equalizing context windows should be reported before treating the 15/6/9 taxonomy and the 10/30 coverage of the no-training rule as robust.","section":null},{"comment":"§IV-D and Fig. 4: the load-bearing deployment rule n_train < 700 and S ≥ 0.05 is acknowledged as cluster-derived (to include ILI at 579 with a gap to the next cluster). With only 30 datasets and LOO accuracy 53–60%, the thresholds are not validated out-of-sample. Either reframe the rule as a descriptive observation on this suite, or provide leave-one-out / hold-out stability of the 700/0.05 cutoffs and of the 10/30 resolution claim.","section":null}],"minor_comments":[{"comment":"Table III mean MASE is dominated by a few high-MASE series (ETTh2, Exchange, Pedestrian, etc.); the paper already prefers “Best on k”—make that the primary ranking in the table body, not only a note.","section":null},{"comment":"Exchange MASE caveat (m=5, S=0.00) and Rideshare COVID-padding removal should be more prominent in Table I / §III-C so readers do not over-interpret those two FM-dominant labels.","section":null},{"comment":"Full fine-tuning is omitted from main tables for instability; a short appendix table of catastrophic-failure rates would make the LoRA-default recommendation easier to audit.","section":null},{"comment":"Figure 1 is dense; consider splitting FM-dominant vs break-even panels or adding a compact summary table of n* ranges next to the figure.","section":null},{"comment":"Related-work citations with “?” placeholders (Zeng et al., Ye et al.) should be completed before camera-ready.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core empirical map is useful and the code release is a real strength; the main risk is marketing language (“unconditionally justified,” “resolves 10 of 30 immediately”) that outruns the best-of / discrete-grid / post-hoc-threshold protocol. A revision that recenters on fixed-model, mean-seed results and treats the 700/0.05 rule as descriptive would make this a solid empirical contribution for a methods/applications venue. Scope fit is good for a forecasting or applied ML journal; less so for a pure theory venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is the first systematic data-volume break-even study of time-series foundation models against classical baselines. What is new is the framing: not “which model wins at full data,” but at what n classical methods catch zero-shot Chronos/Moirai/Lag-Llama, plus a concrete two-step deploy rule and released code.\n\nThey ran a large, clean grid—30 Monash-style datasets, six fractions, multiple FM settings, four classical methods, three seeds, fixed-denominator MASE, chronological splits. The scaling curves and Table II make the heterogeneity visible: FM zero-shot wins at every fraction on 15/30; classical wins early (as little as 2%) on 6; the rest need substantial data. The short-series observation that LoRA can hurt (ILI, M4-Hourly) is useful and under-discussed. XGBoost is treated fairly. Citations to Chronos, Moirai, Lag-Llama, FFORMS/FFORMA, and Monash are appropriate; the math is operational, not circular.\n\nSoft spots are real but proportional. The headline “GPU unconditionally justified on 15” and the n_train<700 & S≥0.05 rule rest on best-of-FM vs best-of-classical and min-over-seeds MASE. The authors themselves note that a single-family deployer will see higher n*, flag nine gaps <0.3 MASE that may reverse with more seeds, and admit the 700/0.05 cutoffs are cluster-derived on this suite (LOO only 53–60%). FM always gets full history while classical is fraction-restricted, so the comparison is slightly asymmetric. Full fine-tuning is unstable and rightly sidelined. None of this overturns the map; it means the marketed “resolves 10/30 with no training” is a useful heuristic, not a validated boundary.\n\nThis is for applied forecasting people deciding whether to stand up GPU infrastructure, and for anyone tired of fixed-size FM leaderboards. It deserves a serious referee. I would bring it to reading group, cite the taxonomy and the LoRA-on-short-series warning, and treat the exact thresholds as provisional pending larger benchmarks and mean±error reporting.","headline":"Useful first break-even map of time-series FMs vs classical methods; the 15/30 FM-dominant count and n<700 rule are real but overstated by best-of-model selection and seed-sensitive margins.","tokens_in":19382,"tokens_out":563,"would_cite":true,"duration_ms":5120,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"On half of 30 forecasting benchmarks, zero-shot foundation models beat classical methods at every data size; a short-series seasonality rule decides ten cases without any training.","keywords":["time series forecasting","foundation models","break-even analysis","model selection","scaling laws","zero-shot","LoRA fine-tuning","XGBoost"],"falsifier":"Re-run the same six-fraction protocol on an independent suite of fifty-plus datasets spanning new domains; if the length-under-700 and seasonality-at-least-0.05 rule no longer correctly identifies zero-shot foundation-model wins, or if many of the fifteen FM-dominant cases reverse with more seeds or denser grids, the claimed deployment rule fails.","tokens_in":19192,"feed_emoji":"📈","tokens_out":748,"duration_ms":6878,"temperature":0.7,"pith_summary":"Practitioners must decide whether the GPU cost of a time series foundation model is worth it versus classical methods such as XGBoost. This paper answers by measuring break-even: the training volume at which classical methods first match or beat zero-shot foundation models. Across thirty datasets and six training fractions, foundation models win at every fraction on fifteen datasets, so GPU use is justified regardless of data volume. On six datasets classical methods already win with as little as two percent of the data (as few as twenty-one samples); the remaining nine cross later. One rule needs no training at all: when training length is under seven hundred samples and seasonal strength is at least 0.05, use the foundation model zero-shot and skip fine-tuning; that rule settles ten of the thirty decisions immediately. LoRA fine-tuning often fails to help and can hurt short series. The authors turn the findings into a two-step checklist—check length and seasonality, then run a five-to-ten-percent pilot only if needed—so teams can choose before they buy infrastructure.","feed_headline":"When GPU time series models beat XGBoost—and when they don’t","feed_subtitle":"On 15 of 30 datasets they win at every data size; a short-series rule settles 10 cases with no training.","key_machinery":"Break-even point n*: the smallest training size at which any classical method first matches or beats the best zero-shot foundation model; datasets are then partitioned into FM-dominant (no finite n*), early, mid, and late regimes, and a length-plus-seasonality rule is extracted for deployment.","core_discovery":"The relative value of pretrained time series foundation models versus classical forecasters is data-dependent and can be summarized by a break-even point: on fifteen of thirty benchmarks the foundation model never loses at any training fraction, on six classical methods already win at two percent of the data, and on the rest the crossing lies between twenty-four and eight thousand samples. One empirically robust rule is that if training length is under seven hundred and seasonal strength is non-negligible, zero-shot foundation models are the safe default and fine-tuning should be skipped.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["On 15 of 30 datasets, time series FMs beat classical methods at any data size","Break-even for pretrained forecasters vs XGBoost: 0 to 8,361 samples","If n_train < 700 and seasonal, zero-shot FM wins—skip fine-tuning","GPU foundation models never lose on half the benchmarks vs XGBoost","Classical methods beat zero-shot FMs at 2% data on 6 of 30 datasets"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The practical cutoffs (training length under seven hundred and seasonal strength at least 0.05) and the discrete fraction grid are tuned to this thirty-dataset suite and may not transfer cleanly to new domains or denser sampling of data sizes.","fun_headline_variants_meta":{"raw":{"variants":["On 15 of 30 datasets, time series FMs beat classical methods at any data size","Break-even for pretrained forecasters vs XGBoost: 0 to 8,361 samples","If n_train < 700 and seasonal, zero-shot FM wins—skip fine-tuning","GPU foundation models never lose on half the benchmarks vs XGBoost","Classical methods beat zero-shot FMs at 2% data on 6 of 30 datasets"]},"model":"grok-4.5","effort":"low","cost_usd":0.007206,"raw_usage":{"total_tokens":1881,"prompt_tokens":924,"num_sources_used":0,"completion_tokens":120,"cost_in_usd_ticks":72060000,"prompt_tokens_details":{"text_tokens":924,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":837,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":924,"tokens_out":120,"duration_ms":6084,"temperature":1.0,"reasoning_tokens":837,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T11:28:02.611711+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same six-fraction protocol on an independent suite of fifty-plus datasets spanning new domains; if the length-under-700 and seasonality-at-least-0.05 rule no longer correctly identifies zero-shot foundation-model wins, or if many of the fifteen FM-dominant cases reverse with more seeds or denser grids, the claimed deployment rule fails.","supporting_citations":[],"review_version":1}