{"id":"d37bc7f3-387e-471a-81c0-11bf031d0663","arxiv_id":"2506.21570","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pretrained T5 language weights give a persistent validation-loss advantage over random initialization for low-data time series forecasting, and the advantage does not vanish within the training budget.","lead":"This paper tests whether language model weights help train time series forecasters, and finds they beat random initialization across tokenizers and model sizes. The authors also claim the advantage never disappears no matter how much data a from-scratch model sees, which would contradict earlier scaling laws.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'non-vanishing transfer gap' rests on reading finite training curves as asymptotic convergence; the random-init plateaus are not established as true convergence, so the central claim is under-supported.","rationale":"The reader's CONDITIONAL verdict correctly identifies the weakest assumption: random-init plateaus are treated as convergence without evidence. My stress-test converges on the same point but sharpens it: the effective-transfer metric's vertical asymptotes are a direct consequence of defining inverse losses only over the observed training range, so they cannot by themselves establish an infinite data requirement. This is the single most load-bearing issue because the paper's headline novelty over Hernandez et al. (2021) is precisely the claim of a non-vanishing, asymptotic transfer gap. If the random-init curves are merely slow, the headline conclusion collapses to a finite-budget observation. The proposed concrete test—extended training with schedule restarts plus convergence diagnostics—directly targets this assumption. The paper's other contributions, such as tokenizer comparisons and the Flan-T5 underperformance finding, are not threatened by this concern. Since the reader already conditioned acceptance on convergence evidence, my assessment does not change the verdict; it reinforces the condition and specifies how to satisfy it.","tokens_in":7702,"tokens_out":2696,"duration_ms":36704,"concrete_test":"Run the same 220M T5 experiments with the bin tokenizer but extend training to at least 5-10x the current token budget, using a constant or cyclically restarted learning rate after the random-init curve appears to plateau. If the random-init validation loss resumes decreasing and approaches the pretrained-init curve, the non-vanishing gap is an artifact of the finite training budget; if the loss remains flat across the extended budget and the pretrained-init curve continues to decrease, the asymptotic claim is supported. Report loss derivatives or gradient norms at the end of both regimes to distinguish true convergence from schedule-induced freezing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Contribution 2 and the abstract claim that 'no amount of additional tokens' lets a randomly initialized model match a pretrained-LM initialization is an asymptotic statement, but the evidence is entirely finite-budget training curves. In Section 3 and Figures 1-3, random-init validation losses flatten 'earlier' and at a higher value, and the effective-transfer metric D_T(ℓ) in Appendix D exhibits vertical asymptotes because L_R^{-1}(ℓ) is undefined for target losses below the observed random-init floor. This only shows that the random-init model did not reach those loss levels within the tested token budget, not that it cannot. Transformer training from scratch is known to exhibit long plateaus before later improvement, and the paper does not report convergence diagnostics such as loss derivatives, gradient norms, or extended training with adjusted schedules. The hyperparameter sweep in Appendix B varies LR, batch size, weight decay, and warm-up, but it does not vary total training duration or use learning-rate restarts after apparent convergence. Therefore the central claim that the transfer gap is non-vanishing is an extrapolation, not a demonstrated result. The weaker empirical claim that pretrained init beats random init in the tested low-data regime is much better supported and likely correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether initializing a T5 encoder-decoder with pre-trained language-model weights helps when fine-tuning on univariate time series forecasting. The authors compare random, T5-pretrained, and Flan-T5 initializations across three tokenizers (naive, lag, bin) and three model sizes, and they define an 'effective transfer' metric based on the difference in training tokens needed to reach a given validation loss. Their central claim is that the validation-loss advantage of pre-trained initialization is non-vanishing: no amount of additional training tokens would let a randomly initialized model match a pre-trained initialization, contrary to the transfer-gap scaling law of Hernandez et al. (2021). They also report that instruction-tuned Flan-T5 underperforms the purely pre-trained T5 initialization for this task.","tokens_in":7943,"tokens_out":7693,"duration_ms":79293,"significance":"If the non-vanishing transfer gap were established, it would be an interesting counterexample to the vanishing-gap scaling law of Hernandez et al. (2021) and would suggest that language pretraining imparts an asymptotic inductive bias for time series modeling. The paper's controlled comparison of tokenizers, initializations, and model sizes, along with the proposed effective-transfer metric, is a useful empirical contribution. The authors are appropriately cautious in the conclusion about the limited T5 family and about distinguishing 'more data' from 'structural similarity.' However, the headline asymptotic claim is not supported by the presented finite-budget training curves, so the paper currently overstates its main contribution.","major_comments":[{"comment":"The central claim that 'no amount of additional tokens' would let a randomly initialized model match a pre-trained-language initialization is an asymptotic statement, but the evidence consists entirely of finite-budget training curves. The randomly initialized models are described as having 'converged at a higher loss value and earlier,' yet the paper does not report the total number of training tokens, a convergence criterion, loss derivatives, gradient norms, or any learning-rate restarts. Transformer training from scratch is known to exhibit long plateaus before later improvement, so the observed plateau does not establish true convergence. The vertical asymptotes in Figure 2(b) and the effective-transfer definition in Appendix D occur precisely when L_R^{-1}(ℓ) is undefined because the randomly initialized model never reached that loss level within the tested token budget; this only shows that the random model did not reach that loss in the experiment, not that it cannot. Please either add long-horizon training with schedule restarts and convergence diagnostics, or explicitly reframe the contribution as a finite-budget statement and soften the claimed contradiction with Hernandez et al. (2021).","section":"Section 3, Figures 1-3; Appendix D"},{"comment":"The hyperparameter comparison is not a sufficient control for the asymptotic claim. Appendix B sweeps learning rate, batch size, weight decay, and warm-up duration, but it does not vary total training duration or use learning-rate restarts after apparent convergence. Since the paper's headline is about whether random initialization can eventually catch up, fixing a single budget and visually declaring convergence is not a fair test of that question. In addition, the main figures (Figures 1-3) show no error bars or per-seed variability despite the statement that runs used three random seeds; Appendix B reports standard deviations only for the hyperparameter sweep. The paper should report seed variability in the main comparisons, at least for the figures that support the 'holds across design choices' claim.","section":"Section 2 and Appendix B"},{"comment":"The effective-transfer metric D_T(ℓ) is defined through inverse functions L_R^{-1}(ℓ) and L_P^{-1}(ℓ), which require the validation-loss curves to be monotonic in training tokens. The paper does not establish monotonicity, and the Flan-T5 curve is explicitly described as 'much noisier and spikier' in Section 3. Without a stated procedure for inverting non-monotonic curves (e.g., fitting a smooth monotone curve and taking the first hitting time), the metric is not well-defined for all loss levels and the quantitative claims in Figure 2(b) are not reproducible. Please specify the inversion procedure and restrict D_T to loss levels at which the inverse is unambiguous.","section":"Appendix D"}],"minor_comments":[{"comment":"The caption says 'lower loses'; this should be 'lower losses.'","section":"Figure 2(a)"},{"comment":"The lag tokenizer dimension is not stated; given s_t = [x_t, x_{t-ℓ_1}, ..., x_{t-ℓ_p}]^T, d_token = p+1, and the lag set L used in the experiments should be specified.","section":"Section 2.2"},{"comment":"The paper refers throughout to 'validation loss' but never defines the loss function used for evaluation; please state whether it is the negative log-likelihood of the predictive distribution and describe the validation split.","section":"Section 2"},{"comment":"The sentence 'This behavior contrasts with Hernandez et al. (2021)' should be reconciled with the later sentence 'Consistent with Hernandez et al. (2021)' in the same section; the paper should clarify that the contrast concerns the vanishing of the transfer gap with data, not the model-size dependence.","section":"Section 3"},{"comment":"The exact pre-trained checkpoints are not specified (e.g., specific T5-Efficient and Flan-T5 model identifiers); please list them for reproducibility.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical finding (pre-trained beats random in the low-data regime) is plausible and likely correct, but the asymptotic 'can't catch up' claim is oversold. I would not recommend rejecting, but the revision must either add substantial evidence (longer training, schedule restarts, convergence diagnostics, error bars) or weaken the claim to a finite-budget statement. The paper would be stronger if it reported the training budget and seed variance. There is no circularity problem in the comparison itself; the citation to Hernandez et al. is apt, but the comparison should acknowledge that the downstream data budget here is much smaller than in scaling-law transfer studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical paper with one overclaimed headline. The systematic comparison is real work: T5 small/base/large, three tokenizers, three initialization types, and a clean external baseline against random init. The observation that plain T5 transfers better to time series than instruction-tuned Flan-T5 is interesting and not in the prior work they cite. The effective-transfer metric is a reasonable way to quantify data savings, and it is defined from the measured loss curves rather than fitted to a conclusion. The weak claim — pretrained beats random initialization in the tested low-data regime — is well supported, and it holds across tokenizers and sizes. I'd trust that.\n\nThe soft spot is the strong claim. The abstract and Contribution 2 say no amount of additional tokens lets a randomly initialized model match the pretrained initialization. That's an asymptotic statement, and the evidence is entirely finite-budget training curves. Random-init losses flatten early, but the paper never shows they have truly converged: no loss derivatives, no gradient norms, no extended training with restarts, no stated training duration in the main text. Error bars are missing from the main figures. The vertical asymptotes in the effective-transfer metric (Appendix D) appear precisely because the random-init curve never reaches the lower loss levels within the tested budget — which is exactly the kind of artifact that makes a finite-budget claim look like an infinite one. So the headline is under-supported. The authors may be right, but this version doesn't demonstrate it.\n\nMinor things: the Flan-T5 result rests on a single model family, so keep it as a suggestive datapoint, not a general law. The paper says code and checkpoints will be released upon acceptance; for a paper whose claim depends on training dynamics, that should be a condition, not a promise. The self-citations are appropriate — Chronos, Lag-Llama, and the EEG work are the direct prior art.\n\nWho's this for? Practitioners deciding whether to reuse a pretrained LM for forecasting in low-data settings, and people studying transfer scaling. It deserves a serious referee, but the referee should ask for convergence evidence, error bars, training lengths, and artifacts before the asymptotic claim can stand. I'd also ask the authors to soften the claim to a finite-budget gap plus a conjecture.","headline":"Solid empirical core — pretrained T5 beats random init for time series in the low-data regime — but the non-vanishing transfer gap is an extrapolation from finite training curves, not a demonstrated result.","tokens_in":8512,"tokens_out":2428,"would_cite":true,"duration_ms":26908,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No amount of training data lets a randomly initialized model match a pretrained-language initialization on time series forecasting.","keywords":["transfer learning","time series forecasting","language model initialization","T5","tokenization","scaling laws","effective transfer","low-data regime"],"falsifier":"Extend randomly initialized T5 training on the same LOTSA data to several times the current token budget with a wider hyperparameter search; if the validation loss resumes a smooth decrease and eventually reaches or crosses the pretrained-initialized models' loss, the claimed permanent transfer gap is refuted.","tokens_in":7471,"feed_emoji":"📈","tokens_out":6742,"duration_ms":61823,"temperature":0.7,"pith_summary":"Fine-tuning a T5 text-to-text transformer on univariate time series gives better validation loss when the backbone starts from pretrained language weights rather than random weights, and the paper claims this advantage is permanent. Across three tokenizers (raw values, lag vectors, and bins) and three backbone sizes (60M, 220M, 770M), the pretrained-initialized models eventually improve smoothly while the randomly initialized models plateau. The paper defines an 'effective transfer' measure—how many extra training tokens a random model would need to match the pretrained model's loss—and finds it grows without bound. This contradicts the scaling-law expectation, from Hernandez et al. (2021), that the transfer gap should vanish as relevant training data increases. If correct, the result means language pretraining changes the loss landscape itself, not just the starting point.","feed_headline":"Random starts can't catch language-pretrained models on time series","feed_subtitle":"The transfer gap does not shrink as training data grows, contradicting scaling-law predictions.","key_machinery":"The central object is the effective transfer measure $D_T(\\ell) = L_R^{-1}(\\ell) - L_P^{-1}(\\ell)$, the difference in the number of time series training tokens needed for a randomly initialized model and a pretrained-language-initialized model to reach the same validation loss $\\ell$. When the random model's loss converges first, the random inverse curve hits a floor and $D_T(\\ell)$ blows up, making the non-vanishing transfer gap precise. The argument is carried by controlled comparisons of three backbone initializations (random, pretrained T5, instruction-tuned Flan-T5), three tokenizers (naive, lag, bin), and three T5 sizes, trained on the LOTSA dataset with a shared output head from Chronos-style bin forecasting.","core_discovery":"The central claim is that the benefit of initializing a time series forecaster from pretrained language weights is asymptotic, not just a finite-data head start. The paper shows that on the LOTSA training distribution, T5 models fine-tuned from T5-Efficient weights keep decreasing in validation loss long after identically sized models trained from random weights have converged, so the gap between the two curves does not close as more time series tokens are seen. The paper quantifies this as effective transfer, the extra data a randomly initialized model would need to reach a given loss level, and reports vertical asymptotes in this quantity—the formal symptom of a transfer gap that no amount of additional tokens can close. It further reports that instruction-tuned Flan-T5 initialization transfers worse than pretraining-only T5, despite being better on language tasks, and that larger pretrained backbones help while larger random backbones hurt.","pith_inferences":["If the transfer gap is truly asymptotic for T5, a natural next test is whether it appears for other language backbones and for other continuous-signal modalities such as EEG or sensor data.","Because the random models' plateau is the load-bearing observation, the paper's claim would be strengthened by training random initializations for several times longer and with a wider hyperparameter sweep to show the plateau is not a training-budget artifact.","The success of bin tokenization suggests the underlying advantage may be alignment with the discrete-input inductive bias of language models; one could test this by comparing against an encoder pretrained on unrelated discrete sequences rather than natural language."],"forward_implications":["Pretrained-language initialization lowers validation loss for time series forecasting compared with random initialization across all tested tokenizers and backbone sizes in the low-data regime.","The advantage persists at the end of training, so the benefit is not simply a faster start but a lower asymptotic loss curve.","Upstream instruction tuning does not help transfer: Flan-T5 underperforms the pretraining-only T5 despite stronger language performance, indicating that better language scores do not imply better time series transfer.","Larger pretrained backbones improve downstream loss, while larger randomly initialized backbones worsen it, so scale and initialization interact.","Bin tokenization, which turns forecasting into a classification problem over quantized values, gives the smoothest convergence and lowest loss among the tokenizers tested."],"supporting_citations":[{"why":"The scaling law the paper contradicts; it predicts the transfer gap should vanish as relevant training tokens increase.","marker":"Hernandez et al. (2021)"},{"why":"Supplies the bin tokenization and output distribution used to train the forecasting models.","marker":"Ansari et al. (2024)"},{"why":"Prior evidence that T5 transfers to EEG time series, motivating the choice of T5 as the backbone.","marker":"Bayazi et al. (2024)"},{"why":"Source of the T5-Efficient pretrained-only initialization weights.","marker":"Tay et al. (2022)"},{"why":"Source of the instruction-tuned Flan-T5 initialization and of the language-task performance comparison.","marker":"Chung et al. (2022)"},{"why":"Provides the LOTSA dataset used to train all models.","marker":"Woo et al. (2024)"},{"why":"Introduces lag tokenization, one of the three tokenization schemes compared.","marker":"Rasul et al. (2024)"},{"why":"Framework for studying pretrained transformers as general engines, including the scaling-with-size behavior the paper extends to time series.","marker":"Lu et al. (2021)"}],"fun_headline_variants":["Random init can't close gap even with infinite data","Language pretraining gives uncatchable lead on time series","Transfer gap is asymptotic, random starts plateau early","Scaling laws defied: LM transfer beats random init always"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the randomly initialized models have truly converged when their validation loss flattens, rather than only slowing down within the training budget used; if longer training or better hyperparameters would make them improve again, the non-vanishing transfer gap could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Random init can't close gap even with infinite data","Language pretraining gives uncatchable lead on time series","Transfer gap is asymptotic, random starts plateau early","Scaling laws defied: LM transfer beats random init always"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1592,"prompt_tokens":877,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":493,"tokens_out":715,"duration_ms":8768,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:11:47.749744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Extend randomly initialized T5 training on the same LOTSA data to several times the current token budget with a wider hyperparameter search; if the validation loss resumes a smooth decrease and eventually reaches or crosses the pretrained-initialized models' loss, the claimed permanent transfer gap is refuted.","supporting_citations":[{"cited_title":"Scaling laws for transfer, 2021","cited_arxiv_id":null,"evidence_quote":"The scaling law the paper contradicts; it predicts the transfer gap should vanish as relevant training tokens increase."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior evidence that T5 transfers to EEG time series, motivating the choice of T5 as the backbone."}],"review_version":1}