{"id":"a97dfe09-1588-4c46-8b34-f5bfe2e77016","arxiv_id":"2608.10149","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fine-tuned 1.7B language model routes time series forecasting ensembles by reasoning over hybrid text-number series features and retrieved similar cases, beating fixed and learned ensemble baselines on eight benchmarks.","lead":"This paper presents REATS, an LLM-based router that assigns ensemble weights to time series forecasting models by reasoning over textual and numerical descriptions of each series. It reports lower forecast error than fixed, neural, and zero-shot LLM ensemble baselines on eight benchmarks, with natural language explanations for each weighting decision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG prompt injects oracle-derived weights at inference; without a retrieval-only baseline, the LLM-specific contribution to REATS' gains is unestablished.","rationale":"The reader's weakest assumption concerned the stability and transfer of training-set oracle weights. My concern is adjacent but sharper: those oracle weights are not only training labels; they are injected into the prompt at inference through the RAG prior, creating a possible shortcut. This is load-bearing because the paper's headline is about LLM reasoning. If a trivial retrieval baseline achieves comparable MSE, then the experiments do not support the claimed mechanism even if the numbers are correct. The paper's own w/o RAG ablation confirms that the prior is important, making the missing control especially salient. I am not alleging any leakage of test labels: the RAG pool is training-set-only, and the zero-shot baselines share the same prompt components. The issue is that the fine-tuned model is explicitly trained to imitate oracle weights and to reason about \\bar{w}, so the comparison to zero-shot LLMs does not control for the retrieval prior. The proposed check is low-cost and would settle the question: if REATS substantially beats RAG-only, the central claim is strengthened; if not, the claim needs to be reframed. The reader's conditional verdict seems appropriate; I would keep it conditional but add this baseline as an explicit requirement. Since the verdict category already is CONDITIONAL, I mark UNCHANGED while noting the added condition.","tokens_in":34910,"tokens_out":9767,"duration_ms":105559,"concrete_test":"Run a RAG-only baseline that for each test sample outputs \\bar{w}, the average of the K=3 retrieved training samples' oracle weights, using the same retrieval pool and L2 distance as REATS; also run a ridge regression from the eight feature groups to oracle weights on the training set and apply it to test samples. Compare average MSE against REATS-GRPO on Table 2(b) and Table 3(a). If either baseline reaches within 10% of 0.1080 / 0.1442, or if \\bar{w} itself is within that margin, the headline claim should be weakened to a retrieval-augmented LLM rather than an LLM reasoning router. Report the same for the w/o RAG variant to quantify the marginal value of the LLM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a fine-tuned 1.7B LLM is a better sample-adaptive ensemble router than fixed, neural, and zero-shot LLM baselines. The paper does not currently isolate the LLM's contribution from the retrieval prior. In Section 3.3, the RAG module retrieves K=3 training samples and puts their average oracle weights \\bar{w} directly into the prompt; the SFT example in Appendix A.8 shows that the input includes 'Optimal ensemble weights(%) on the most similar time series.' Thus at inference the model receives oracle-derived weights for similar training samples. Because the model is fine-tuned to imitate oracle weight rows and its CoT explicitly reasons about agreement/divergence with \\bar{w} (Algorithm 2, Step 3), high test accuracy could be inherited from the retrieval pool rather than from LLM reasoning. Table 3(b) ablates RAG only by removing it from REATS, which shows the pipeline depends on the prior, but it does not compare against a nonparametric baseline that directly outputs \\bar{w}. The zero-shot LLM baselines also receive the same RAG references, but they are not fine-tuned to copy oracle rows, so they do not control for this shortcut. Unless a retrieval-only baseline (or a ridge/kNN regressor from the eight feature groups to oracle weights) is evaluated, the causal role of the LLM's reasoning in the reported MSE gains is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"REATS proposes to replace fixed or black-box ensemble rules for time series forecasting with a fine-tuned 1.7B LLM that outputs sample-adaptive ensemble weights from hybrid textual-numerical features, retrieved similar-case priors, and chain-of-thought reasoning. The pipeline constructs rule-based CoT supervision from per-sample oracle weights computed by a QP on training ground truth, formats weights as integer percentage tables, and trains with SFT followed by GRPO using a reciprocal reward mapping that compresses MSE gaps. Experiments on eight univariate forecasting benchmarks with four foundation-model or four small-model candidates report that REATS-GRPO achieves the lowest average MSE in both model groups and in OOD transfer to unseen candidates, while ablations attribute gains to CoT, RAG, table format, and multi-row supervision.","tokens_in":35232,"tokens_out":7246,"duration_ms":74770,"significance":"The framework is timely and the OOD experiments are carefully constructed: candidate descriptions and the RAG pool are rebuilt from training-set predictions and ground truth, so no test-label leakage is apparent. Rule-CoT removes API dependency with comparable quality, and the reciprocal reward analysis in Appendix A.5.3 provides a concrete, testable account of why naive r=-delta compresses GRPO advantages. If the attribution gap described below can be closed, REATS would be a meaningful step toward interpretable LLM-based ensemble routers that adapt to candidate sets without retraining.","major_comments":[{"comment":"The central claim that fine-tuned LLM reasoning drives the MSE gains is not yet established, because the inference prompt already contains oracle-derived weights for the retrieved training samples. The prompt in Appendix A.8 includes 'Optimal ensemble weights(%) on the most similar time series:', and Algorithm 2 Step 3 instructs the CoT to state agreement or divergence from the retrieved average; the model is SFT-trained to imitate oracle rows, so it may simply adjust the retrieved average. The 'w/o RAG' ablation in Table 3(b) removes both retrieval and the weight prior, so it does not control for the prior alone. Please add baselines that (i) output the retrieved average weight directly (retrieval-only) and (ii) regress the eight feature groups onto oracle weights with a ridge or kNN predictor, evaluated in the same ID and OOD settings, and report the margin of REATS over these controls.","section":"Section 3.3, Algorithm 2, Appendix A.8"},{"comment":"No repeated-seed statistics are reported, and the design parameters K, K', k, lambda1/lambda3, and the format penalty appear to be selected on the same eight benchmarks used for the headline comparisons (e.g., Table 4 picks K'=10, Figure 6 picks the default reward coefficients, and Appendix A.5.1 picks k=20). Thus the current evidence cannot separate genuine improvement from selection on the reported test sets. Please provide mean and standard deviation over at least three seeds for REATS and the strongest baselines, and move hyperparameter selection to a validation-based protocol.","section":"Tables 2-6 and Figure 6"},{"comment":"The comparison with DAPO, DrGRPO, GSPO, and SAPO covers only three datasets (Exchange, Electricity, Weather), while the claim that the reciprocal reward mapping outperforms several recent GRPO algorithmic variants is stated generally. Moreover, the variants are only tested with the naive r=-delta reward, so the interaction between optimizer-level and reward-level fixes is not measured. Please extend the comparison to the full eight benchmarks and include a factorial design (naive vs reciprocal reward, with standard GRPO and at least one variant) to support the stated claim.","section":"Section 4.3, Figure 5(d), Table 8"}],"minor_comments":[{"comment":"There is a typo in Section 2: 'The first istraditional machine learning-based methods' should read 'The first is traditional machine learning-based methods'; Table 2(a) also prints 'NNRLM C' instead of 'RLMC'.","section":"Section 2 and Table 2(a)"},{"comment":"The implementation details do not report the GRPO group size G, sampling temperature, learning rate, number of GRPO steps, or training compute; adding these would substantially improve reproducibility.","section":"Section 4.1"},{"comment":"The RAG reference block in the SFT example shows three rows of retrieved weights while the method states K'=10 rows; please clarify whether the display is truncated or whether the retrieval pool stores fewer rows than the supervision table.","section":"Appendix A.8"}],"recommendation":"major_revision","confidential_remarks":"I support the editor in asking for a retrieval-only control and repeated-seed evaluation. The RAG setup does not appear to leak test labels; the problem is attribution of the LLM's causal contribution. The paper is otherwise coherent and the design choices are well motivated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"REATS is a serious piece of applied LLM work for time series ensembling. The authors fine-tune a 1.7B LLM to output per-sample ensemble weights over forecasting models, using a fixed-token hybrid textual-numerical representation, rule-based chain-of-thought, and a reciprocal GRPO reward. The framework is coherent, and the empirical package is unusually complete: eight benchmarks, two candidate pools, OOD generalization to unseen models, and ablations for CoT, RAG, output format, and reward design.\n\nThe genuinely new piece is the combination: LLM as router with interpretable reasoning, plus the reward mapping analysis. The reciprocal reward argument is well made; Appendix A.5.3 gives a quantitative lower bound on reward dispersion, and the comparison with optimization-level GRPO variants (DAPO, DrGRPO, GSPO, SAPO) is a useful data point. The rule-based CoT matching GPT-generated CoT at zero API cost is a practical result. The OOD experiments are thoughtfully constructed with no test-label leakage.\n\nThe main soft spot is attribution. The RAG module retrieves K=3 training samples and puts the average oracle weights directly into the prompt. The ablation only removes RAG from the full pipeline; it does not compare against a nonparametric baseline that simply outputs the retrieved average. The zero-shot LLMs also receive RAG references, but they are not fine-tuned to copy oracle rows, so they do not control for this shortcut. On reading the paper, I think this is a real concern, not a fatal one: the fine-tuned model still has to parse features and reconcile with the prior, and the SFT-to-GRPO gains show the model is doing something beyond copying. But the causal role of LLM reasoning is not fully established. A retrieval-only baseline and a ridge/kNN regressor from the eight feature groups to oracle weights would settle it.\n\nOther soft spots are more standard: no error bars or multiple seeds, and hyperparameters k, K, K', and lambda are selected on the same benchmarks. That limits the precision of the headline numbers but does not undercut the overall direction. Code and data release would help verification.\n\nWho is this for? Anyone working on LLM-based time series, ensemble methods, or GRPO rewards for regression. It deserves a serious referee. I would want the missing baseline and error bars before trusting the central claim, but the paper is worth engaging with and will likely influence the subfield.","headline":"A solid, well-engineered LLM ensemble router for time series with a thoughtful reward design; the main unresolved question is how much of the gain comes from the LLM rather than the retrieval prior.","tokens_in":35738,"tokens_out":1584,"would_cite":true,"duration_ms":16068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1.7B-parameter language model, fine-tuned with rule-based reasoning and a reciprocal reward, assigns per-sample ensemble weights that beat fixed, neural, and zero-shot LLM baselines on all eight forecasting benchmarks.","keywords":["time series forecasting","ensemble learning","large language models","chain-of-thought reasoning","GRPO","reward design","sample-adaptive weighting","out-of-domain generalization"],"falsifier":"Train REATS-GRPO with oracle weights replaced by validation-set weights or by random but diverse weights on the same benchmarks; if the reported margin over the best baselines does not collapse, the advantage does not depend on the oracle-quality assumption. A second decisive probe is to fit a non-reasoning regressor (for example gradient-boosted trees) on the same eight feature groups and RAG statistics; if it matches or beats REATS-GRPO, the active ingredient is the features, not LLM reasoning.","tokens_in":34731,"feed_emoji":"📈","tokens_out":6824,"duration_ms":59297,"temperature":0.7,"pith_summary":"This paper claims that a 1.7-billion-parameter language model, fine-tuned in two stages, can act as a sample-adaptive ensemble router for time series forecasting: given a hybrid textual–numerical description of a series and a set of candidate forecasters, it produces per-sample ensemble weights that outperform fixed weighting, neural-network routers, and zero-shot LLM prompting on all eight benchmarks tested. The authors argue that the key enablers are a fixed-token representation of the series, rule-generated chain-of-thought reasoning that needs no API calls, diverse multi-row weight supervision, and a reciprocal reward mapping that adapts GRPO to a continuous unbounded regression objective. If the claim holds, LLM reasoning becomes a practical and interpretable tool for model combination, including transfer to unseen candidate models.","feed_headline":"LLM router beats classic ensembles on all 8 forecast benchmarks","feed_subtitle":"A 1.7B LLM reasons over text and numbers to weight forecasters per sample, generalizing to unseen models.","key_machinery":"The machinery is the combination of a structured prompt and a reciprocal reward mapping. The structured prompt renders the series as fixed-token hybrid text (eight feature groups as key-value pairs) together with retrieved similar-sample priors, which makes rule-based chain-of-thought construction possible without API calls. The reciprocal reward mapping $r = 1/(1+k\\delta)$ with $k=20$ compresses the unbounded MSE gap $\\delta$ into $[0,1]$ with polynomial decay, preserving near-oracle sensitivity after GRPO's advantage normalization, unlike the naive $r=-\\delta$ whose outlier-dominated variance collapses the normalized advantages of near-optimal candidates. A ten-row integer-percentage weight table (one oracle row plus nine diverse Dirichlet-sampled rows) provides multi-row supervision and denser rewards during both SFT and GRPO.","core_discovery":"On the paper's own terms, the central discovery is that an LLM router, REATS, assigns ensemble weights by reasoning over eight feature groups of the time series (stationarity, noise, trend, seasonality, autocorrelation, statistics, outliers, distribution) plus retrieved similar training samples, and this beats the strongest traditional baselines: average MSE of 0.1384 on foundation-model candidates versus 0.1597 for optimal validation-set weighting, and 0.1080 versus 0.1352 on small-model candidates, winning on all eight datasets in each group. The same router, trained only on small candidates, generalizes to unseen foundation models (0.1442 vs. 0.1564 best baseline). The paper attributes the gain to two-stage fine-tuning in which SFT imitates rule-based reasoning and GRPO then directly optimizes a reciprocal-transformed MSE reward, with the diverse ten-row weight table supplying denser reward signals.","pith_inferences":["The authors do not test, but the same router design should apply to other continuous model-selection problems (e.g., per-sample classifier blending or hyperparameter recommendation) whenever an offline oracle can be computed from training data.","Because the input token cost is fixed regardless of series length, the approach should scale to very long histories without truncation; this is a direct consequence the authors note but do not test at length 512 in the main results.","A decisive comparison the authors do not run: a non-LLM regressor on the same eight feature groups would isolate whether LLM reasoning or feature engineering drives the gain.","The reciprocal mapping's benefit is predicted to grow as GRPO rollout groups become more mixed in quality; measuring within-group reward variance under $r=-\\delta$ versus $r=1/(1+k\\delta)$ would test the authors' mechanism directly."],"forward_implications":["Sample-adaptive ensembling can be driven by explicit reasoning rather than fixed rules, and the weights are explainable in natural language.","Rule-based chain-of-thought, generated without paid API calls, is sufficient for SFT and is critical for out-of-domain generalization to unseen candidate models.","A small 1.7B fine-tuned LLM outperforms zero-shot much larger LLMs on this routing task, so scale is not the decisive factor.","The reciprocal reward mapping makes GRPO applicable to continuous regression objectives, outperforming optimization-level GRPO variants (DAPO, DrGRPO, GSPO, SAPO) that keep the naive unbounded reward.","The router transfers across datasets and to unseen candidate model sets, suggesting it learns semantic temporal reasoning rather than memorizing model identities."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that the second fine-tuning stage adapts to continuous MSE rewards.","marker":"[10]"},{"why":"RLMC, the neural-network dynamic-ensemble baseline that REATS must outperform.","marker":"[9]"},{"why":"DAPO, a recent GRPO variant compared as an optimization-level alternative to the reciprocal reward mapping.","marker":"[15]"},{"why":"DrGRPO, another GRPO variant used as a baseline in the reward-design comparison.","marker":"[16]"},{"why":"GSPO, a group-sequence policy optimization variant used as a baseline.","marker":"[17]"},{"why":"SAPO, a soft adaptive policy optimization variant used as a baseline.","marker":"[18]"}],"fun_headline_variants":["1.7B LLM reasons to pick best forecast mix per sample","LLM ensemble router: per-sample weights beat static optimum","Reasoning LLM transfers across model families for forecasting","LLM router beats static weighting on 8 benchmark datasets","Per-sample LLM weighting outperforms fixed rules on 8 sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that per-sample oracle weights computed from training-set ground truth are stable and informative enough that a model can learn a mapping from the eight feature groups to those weights and have it transfer to test samples; if the oracle quadratic program overfits training noise, both the SFT labels and the GRPO rewards inherit that bias.","fun_headline_variants_meta":{"raw":{"variants":["1.7B LLM reasons to pick best forecast mix per sample","LLM ensemble router: per-sample weights beat static optimum","Reasoning LLM transfers across model families for forecasting","LLM router beats static weighting on 8 benchmark datasets","Per-sample LLM weighting outperforms fixed rules on 8 sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001586,"raw_usage":{"total_tokens":6351,"prompt_tokens":998,"completion_tokens":5353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":5268}},"tokens_in":614,"tokens_out":5353,"duration_ms":33028,"temperature":1.0,"reasoning_tokens":5268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:34.737651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train REATS-GRPO with oracle weights replaced by validation-set weights or by random but diverse weights on the same benchmarks; if the reported margin over the best baselines does not collapse, the advantage does not depend on the oracle-quality assumption. A second decisive probe is to fit a non-reasoning regressor (for example gradient-boosted trees) on the same eight feature groups and RAG statistics; if it matches or beats REATS-GRPO, the active ingredient is the features, not LLM reasoning.","supporting_citations":[{"cited_title":"Reinforcement learning based dynamic model combination for time series forecasting","cited_arxiv_id":null,"evidence_quote":"RLMC, the neural-network dynamic-ensemble baseline that REATS must outperform."},{"cited_title":"Understanding r1-zero-like training: A critical perspective","cited_arxiv_id":null,"evidence_quote":"DrGRPO, another GRPO variant used as a baseline in the reward-design comparison."}],"review_version":1}