{"id":"9755a763-ca58-4691-a9a7-1623d527bd2f","arxiv_id":"2608.08010","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reward-based fine-tuning of time series foundation models can collapse predictions away from the true future; steering probability mass into a ground-truth neighborhood reduces that collapse and improves forecasts.","lead":"This paper finds that reinforcement learning used to fine-tune time series forecasting models can push predictions away from the true future values in some regions, a problem it calls 'suboptimal collapse.' It proposes a regularizer that steers the model's probability toward the neighborhood of the true values while keeping some spread, and reports better forecasts on several benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1's collapse mechanism rests on unproven relative-amplification condition (Eq. 7); GTN-R's empirical gains may survive, but the stated theoretical contribution is not established.","rationale":"The reader's weakest-assumption analysis identified Eq. 7 as the load-bearing point of the theoretical argument, and my reading agrees. The proposition is not vacuous as a mathematical statement — it shows that if relative amplification holds, collapse follows — but the condition itself encodes the collapse mechanism rather than deriving it from the RL objective. Since the paper explicitly lists theoretical analysis of suboptimal collapse as a contribution and uses it to motivate GTN-R, this is a genuine weakness. However, the central empirical claim — that adding L_mass and L_uni to the RL objective improves forecasting performance over RL post-training baselines — is supported by a consistent pattern across Tables 1-3, and the theory is not strictly necessary for that empirical claim. Therefore I do not move the verdict: CONDITIONAL remains appropriate, contingent on the authors either deriving Eq. 7 from a concrete policy-gradient update or clearly stating it as a postulated failure mode, and on the requested empirical verification. This is the single most load-bearing concern because it undermines the paper's causal narrative without directly overturning its main empirical result.","tokens_in":14948,"tokens_out":10943,"duration_ms":125496,"concrete_test":"Instrument TS-GRPO and TPO training loops so that whenever an A_k event occurs (no sampled point lies in N_r(G) and the advantage-positive trajectories at time t lie in S), the code logs p_k(S)/p_k(G) before and after the update. Then check whether Eq. 7 holds with η>0 for a meaningful fraction of such events. Alternatively, derive Eq. 7 analytically from the actual advantage-weighted policy-gradient step, including softmax renormalization and baseline effects, for at least one of the implemented RL methods. If the inequality fails systematically, Proposition 1 should be re-labeled as a postulated failure mode, not a derived mechanism, and the paper's theoretical contribution should be weakened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is Proposition 1 in the section 'Theoretical Analysis'. Its conclusion — that low-quality regions can self-reinforce until the ground-truth-neighborhood probability approaches zero — depends on Eq. 7, the 'relative amplification' condition: whenever a low-quality-dominated update occurs, the probability of the low-quality region S grows at least (1+η) times faster than the ground-truth neighborhood G. This inequality is not derived from any policy-gradient update, loss, or normalization scheme; it is asserted as a property of 'RL updates increase the probabilities of relatively better sampled trajectories'. But the event A_k that triggers Eq. 7 already includes the assumption that the positively reinforced trajectories lie in S, and the 'initial sampling difficulty' and 'low-quality-dominance' conditions similarly assume that the collapse mechanism is operative. Consequently, Eqs. 8 and 9 are conditional consistency results: if the amplification inequality holds, then G's probability decays geometrically. The theorem does not establish that standard RL post-training produces such amplification, nor that initial sampling difficulty causes it. The empirical 'validation' in Fig. 2a is also a retrospective correlation: points are selected because they collapsed after training, then shown to have had lower initial ground-truth-neighborhood probability, which is close to a restatement of the selection criterion. This weakness does not by itself refute the empirical claim that GTN-R improves forecasting performance in Tables 1-3, but it does mean the paper's first contribution — identifying and explaining the mechanism of suboptimal collapse — is currently unsupported at the level claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper identifies a phenomenon called suboptimal collapse in reinforcement learning (RL) post-training of probabilistic time series foundation models (TSFMs): during RL post-training, the predictive distribution in some forecast regions shifts away from the ground-truth neighborhood even as overall MSE decreases. The authors propose a mechanism based on initial sampling difficulty and self-reinforcing low-quality updates, formalize it in Proposition 1, and use it to motivate a regularization method, GTN-R, which adds two terms to the RL objective: a neighborhood probability mass constraint and a within-neighborhood uniformity constraint. The method is evaluated by integrating it into TPO, TimeRFT, and their own TS-GRPO across several TSFMs and datasets, with standard and zero-shot forecasting results reported in Tables 1 to 3. The paper also presents visualizations of the proposed mechanism and of the effect of directly adding the ground-truth trajectory to the sampled group.","tokens_in":15292,"tokens_out":2447,"duration_ms":28069,"significance":"If the empirical claim holds, the paper makes a useful practical contribution: a simple, model-agnostic regularization that improves RL post-training of probabilistic TSFMs and appears to preserve exploration better than directly adding the ground-truth trajectory. The experimental scope is broad—multiple TSFMs (Moirai, Toto, Moirai-MoE, UniTS), multiple RL methods (TPO, TimeRFT, TS-GRPO), standard and zero-shot settings—and the reported gains are consistent rather than cherry-picked. The paper also gives credit for identifying a phenomenon (suboptimal collapse) that practitioners may encounter. However, the theoretical analysis as presented does not establish the proposed mechanism: Proposition 1 rests on an assumed inequality that essentially contains the collapse phenomenon, and the empirical support in Fig. 2a is retrospective. The core contribution is therefore currently an empirical regularization technique whose mechanism is plausible but not proven. The absence of variance estimates, the per-dataset tuning of the neighborhood radius, and the lack of code further weaken the empirical claim's verifiability.","major_comments":[{"comment":"The 'relative amplification' condition in Eq. (7) is asserted rather than derived from any policy-gradient update, loss, or normalization scheme. Since Eq. (7) already states that the low-quality region S grows at least (1+η) times faster than the ground-truth neighborhood G, Eqs. (8) and (9) are conditional consistency results: if such an inequality holds, then G's probability decays geometrically. This does not establish that standard RL post-training produces this amplification or that initial sampling difficulty causes it. The theorem should be reframed either as a conditional statement with Eq. (7) explicitly labeled an assumption, or the authors should derive Eq. (7) from the specific RL objectives used (TPO, TimeRFT, TS-GRPO). As written, the theoretical contribution in contribution 1 is not supported.","section":"Theoretical Analysis, Eq. (7)"},{"comment":"The empirical validation of the collapse mechanism is retrospective: the authors first train the model, then select time points whose final ground-truth-neighborhood probability is below 0.001, and then show that these same points had lower initial ground-truth-neighborhood probability. This is close to a restatement of the selection criterion and does not provide independent evidence for the self-reinforcing mechanism. A prospective analysis—for example, fixing points by their initial probability and tracking their evolution, or comparing with a null model where final probabilities are randomized—would be needed to support the claim that initial sampling difficulty drives collapse.","section":"Empirical Analysis, Fig. 2a"},{"comment":"All results are reported as five-seed averages without standard deviations, confidence intervals, or significance tests. Given that many reported improvements are small (e.g., Table 1, TPO+GTN-R vs. TPO on several datasets differs by 0.01–0.02 MSE), the reader cannot assess whether the improvements are statistically meaningful. Please report per-seed results or variance estimates, and where appropriate perform paired tests across seeds or evaluation horizons.","section":"Experiments: Tables 1 to 3"},{"comment":"The neighborhood radius r is tuned per dataset (1 for Loop Seattle, 100 for ENTSO-e Load, 0.1 otherwise) and selected 'based on the hyperparameter sensitivity results,' but the main text does not report those results or a selection procedure. Because the mass and uniformity regularizers operate on the interval [y_t−r, y_t+r], the radius directly controls the strength of the regularization, and per-dataset tuning raises the question of whether the reported gains are robust to the choice of r. Please report the sensitivity analysis and, if possible, use a principled or fixed rule for r.","section":"Implementation Details"},{"comment":"No code or training scripts are provided, and the paper introduces a new method (TS-GRPO) that is described only in the appendix. Given the dependence on exact RL implementations (advantage estimation, reward clipping, sampling schemes) and the per-dataset hyperparameters, the experiments are not currently reproducible from the manuscript alone. Releasing code and experimental configurations would materially strengthen the empirical claim.","section":"Reproducibility"}],"minor_comments":[{"comment":"The phrase 'Given a train date' appears to be a typo for 'Given a training data' or 'Given a training sample'; please correct.","section":"Overall Pipeline"},{"comment":"The notation '×101' in the sampling-variance panels is confusing; the caption says the actual variances are 10 times the plotted values, but the label reads as a power of ten. Please clarify the scaling notation.","section":"Figures 2 and 5"},{"comment":"The ground-truth reward is clipped at 1.5 times the maximum sampled reward 'to prevent an excessively high ground-truth reward from compromising training stability,' but no analysis or justification is given for this particular clipping factor. Please provide a sensitivity check or a principled rationale.","section":"Empirical Analysis, 'with GT' setting"},{"comment":"The within-neighborhood uniformity constraint uses KL divergence between the normalized predictive distribution and the uniform distribution on the neighborhood; the text does not discuss what happens when the neighborhood probability mass is zero or extremely small, which can make the normalized density undefined. Please state how such cases are handled in the implementation.","section":"Method, Eq. (16)"},{"comment":"Some citations appear to be to future or unpublished work (e.g., Li et al. 2026, Liu et al. 2026, Yan et al. 2026). Please verify that these references are publicly available and correctly dated, or mark them clearly as preprints with arXiv identifiers.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical contribution is potentially publishable, but the current version overclaims the theoretical analysis: Proposition 1 is conditional on an assumption that is very close to the conclusion, and the empirical validation in Fig. 2a is retrospective. The main empirical tables also lack error bars, and the per-dataset radius tuning plus missing code make the results hard to verify. I recommend major revision rather than rejection because the empirical pattern—consistent gains across many settings—is valuable even if the mechanism is not fully established, and the authors can address the theoretical framing and add variance/reproducibility information within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GTN-R is a simple, plausible regularizer for RL post-training of probabilistic time series foundation models, and the empirical work is broad enough to back the main claim. Don't lean on the theory section; it formalizes a conjecture rather than proving a mechanism.\n\nThe paper identifies suboptimal collapse: during RL post-training, the predictive distribution in some regions drifts away from ground truth. The fix is to add two terms to the RL loss — one maximizing probability mass in a neighborhood around ground truth, one encouraging uniformity inside that neighborhood. That's a sensible way to pull the model toward good trajectories without collapsing to behavior cloning. The experiments cover several backbones (Moirai, Toto, Moirai-MoE, UniTS), three RL methods, eight datasets, and zero-shot transfer. Gains are modest but fairly consistent, and the 'with GT' control shows direct ground-truth inclusion hurts generalization, which is a useful sanity check.\n\nMain weakness: Proposition 1. Eq. 7 assumes that when a low-quality-dominated update occurs, the probability of the low-quality region grows at least 1+η times faster than the ground-truth neighborhood. That is basically the collapse mechanism restated, not derived from a policy-gradient update or loss. The paper is candid that this is a possible failure mode, but calling it theoretical analysis overstates it. The empirical validation in Fig. 2a is retrospective: collapsed points are selected after training and shown to have had lower initial ground-truth-neighborhood probability. That is a correlation, not a causal test. Also: no error bars in the main tables, the neighborhood radius is tuned per dataset, and code isn't provided. These are real limits, but they don't refute the central empirical claim that GTN-R helps; they just make the size of the gains uncertain.\n\nWho it's for: anyone doing RL post-training of probabilistic TSFMs, especially in energy, traffic, or weather forecasting. It deserves peer review. The referees should ask for variance estimates, code, and either a real derivation of Eq. 7 from an actual update or a clear statement that the theory is a formalized conjecture.","headline":"A plausible and broadly tested regularizer for RL post-training of TSFMs; the theory is a conjecture dressed as a theorem, but the empirical claim holds up.","tokens_in":15814,"tokens_out":3268,"would_cite":true,"duration_ms":30804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RL post-training of time series foundation models can collapse toward wrong forecasts; this paper adds a ground-truth neighborhood regularizer that prevents it.","keywords":["time series forecasting","foundation models","reinforcement learning post-training","suboptimal collapse","ground-truth neighborhood regularization","probabilistic time series forecasting","neighborhood probability mass","within-neighborhood uniformity"],"falsifier":"Track the ground-truth-neighborhood probability of individual forecast points throughout vanilla RL post-training; a point with high initial $p_0$ that nonetheless collapses below 0.001 would contradict Proposition 1's conditioning on low initial sampling difficulty.","tokens_in":14722,"feed_emoji":"🎯","tokens_out":8050,"duration_ms":74562,"temperature":0.7,"pith_summary":"The paper identifies a failure mode in reinforcement-learning (RL) post-training of probabilistic time series foundation models: in some forecast regions, the model's predictive distribution gradually shifts away from the ground truth even as overall error decreases. It calls this suboptimal collapse and argues it arises from a self-reinforcing loop: when the model rarely samples trajectories near the ground truth, the relatively better sampled trajectories tend to lie in a low-quality region, and RL updates that favor them push probability further away from the truth. To break the loop without turning post-training into behavior cloning, the paper proposes Ground-Truth Neighborhood Regularization (GTN-R), which adds two constraints to the RL objective: maximize probability mass inside an interval around the true future value, and keep the conditional distribution inside that interval close to uniform. Experiments across several foundation models, RL methods, and datasets report that GTN-R mitigates collapse, preserves sampling variance, and improves forecasting over the corresponding RL baselines.","feed_headline":"Regularizer stops RL post-training from drifting forecasts off truth","feed_subtitle":"Adds mass near the ground truth plus inside-neighborhood uniformity, cutting MSE without killing exploration.","key_machinery":"The central machinery is the ground-truth neighborhood $N_r(y_t) = [y_t - r, y_t + r]$, an interval of radius $r$ centered at each true future value, together with two distributional constraints built on it: the neighborhood probability mass constraint $L_{\\text{mass}}$, defined as the mean log-probability mass inside the neighborhood (Eqs. 12–13), and the within-neighborhood uniformity constraint $L_{\\text{uni}}$, defined as the negative KL divergence between the normalized predictive density inside the neighborhood and the uniform density (Eqs. 14–16). Their weighted sum is added to the original RL objective: $L_{\\text{GTN-R}} = L_{\\text{RL}} + \\lambda_m L_{\\text{mass}} + \\lambda_u L_{\\text{uni}}$ (Eq. 17). The mass term supplies the pull toward high-quality regions; the uniformity term preserves the spread that keeps the model exploring. Proposition 1 supplies the theoretical motivation: once a low-quality-dominated update occurs and amplifies a low-quality region relative to the ground-truth neighborhood (Eq. 7), the probability of further such updates grows and the neighborhood probability can tend to zero.","core_discovery":"The paper's central claim is that suboptimal collapse in RL post-training of probabilistic time series foundation models is caused by a self-reinforcing cycle: in forecast regions where the model rarely samples near the ground truth, the relatively better sampled trajectories tend to lie in a low-quality region, and RL updates that reward those trajectories further increase the probability of that region. Proposition 1 formalizes a sufficient version of this cycle under three conditions—initial sampling difficulty, low-quality-dominance, and relative amplification—and shows that if the relative amplification condition holds, the ground-truth-neighborhood probability can decay geometrically toward zero. The proposed remedy, GTN-R, does not put the ground-truth trajectory into the update; it treats the ground-truth neighborhood $N_r(y_t) = [y_t - r, y_t + r]$ as a reference region, maximizes the probability mass inside it, and forces the conditional distribution inside it toward uniform. The paper argues this both counters collapse and preserves exploration, and reports consistent MSE and MAE improvements across Moirai, Toto, Moirai-MoE, and UniTS under TPO, TimeRFT, and TS-GRPO.","pith_inferences":["Beyond the paper: Eq. 7's relative-amplification condition is stated about any RL update, so the same collapse mechanism should appear in RL post-training of other generative models with continuous outputs; GTN-R-style neighborhood constraints would be a natural transfer.","Beyond the paper: the neighborhood radius $r$ is chosen per dataset; tying $r$ to the predictive distribution's dispersion or to local noise levels would make the regularizer adaptive and remove a source of hyperparameter sensitivity.","Beyond the paper: the uniformity constraint acts like an entropy-like penalty restricted to the neighborhood, so GTN-R could be combined with standard entropy-regularized RL to address collapse and global exploration separately.","Beyond the paper: the paper's 'with GT' experiments suggest a broader principle—use ground truth as a reference region, not as a training sample—that could inform other RL fine-tuning pipelines where gold outputs are available but should not be imitated directly."],"forward_implications":["Any RL post-training method that reinforces relatively better sampled trajectories is susceptible to suboptimal collapse in regions where initial sampling near the ground truth is poor; GTN-R can be attached to such a method as a regularizer without changing its update rule.","Because the ground-truth trajectory never enters the update as a fixed high-reward sample, GTN-R should avoid degrading into behavior cloning; the paper's comparisons show its sampling variance stays higher than the direct-ground-truth variant.","The neighborhood mass and uniformity constraints provide a concrete mechanism for the reported MSE and MAE gains: more frequent sampling of high-quality trajectories plus preserved spread inside the neighborhood.","The method extends to zero-shot transfer: Table 3 reports gains when a model post-trained with GTN-R on one dataset is evaluated on another.","The uniformity term is what distinguishes GTN-R from simply adding the ground-truth sample, since it prevents probability mass from concentrating on only a few locations inside the neighborhood."],"supporting_citations":[{"why":"Supplies TPO, one of the RL post-training methods GTN-R is integrated with, and is cited for the claim that RL improves TSFMs by increasing probabilities of relatively better trajectories.","marker":"Qi et al. 2025"},{"why":"Supplies TimeRFT as a comparison method and dataset split source, and is cited for the relative-better-trajectory reinforcement premise underlying Eq. 7.","marker":"Li et al. 2026"},{"why":"Provides the post-training evaluation framework, most baseline results, and dataset splits used in Tables 1 and 3.","marker":"Qiao et al. 2025"},{"why":"Introduces Moirai, the one-shot probabilistic backbone used in the main experiments and in the collapse visualization.","marker":"Woo et al. 2024"},{"why":"Introduces Toto, a token-by-token probabilistic TSFM used as a backbone in the experiments.","marker":"Cohen et al. 2024"},{"why":"Introduces Moirai-MoE, a token-by-token sparse-mixture-of-experts TSFM used as a backbone and in the TimeRFT comparisons.","marker":"Liu et al. 2024a"},{"why":"Provides GRPO, the base algorithm from which the paper's TS-GRPO post-training method is derived.","marker":"Shao et al. 2024"},{"why":"Supplies the Gauss–Legendre quadrature method used to evaluate the neighborhood mass integral and the KL divergence.","marker":"Swarztrauber 2003"}],"fun_headline_variants":["Ground-truth neighborhood regularizer curbs RL collapse in forecasting","RL post-training drift fixed by steering model toward truth","Neighborhood regularization stops suboptimal collapse in TSFMs","GTN-R anchors RL post-training to reality, improving forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theoretical story stands on the relative-amplification condition (Eq. 7): when a low-quality region dominates an update, that region's probability grows at least $(1+\\eta)$ times faster than the ground-truth neighborhood, and this is asserted from the generic behavior of RL rather than derived from the specific update rules.","fun_headline_variants_meta":{"raw":{"variants":["Ground-truth neighborhood regularizer curbs RL collapse in forecasting","RL post-training drift fixed by steering model toward truth","Neighborhood regularization stops suboptimal collapse in TSFMs","GTN-R anchors RL post-training to reality, improving forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1410,"prompt_tokens":1004,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":338}},"tokens_in":620,"tokens_out":406,"duration_ms":5863,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:33:54.854940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the ground-truth-neighborhood probability of individual forecast points throughout vanilla RL post-training; a point with high initial $p_0$ that nonetheless collapses below 0.001 would contradict Proposition 1's conditioning on low initial sampling difficulty.","supporting_citations":[],"review_version":1}