{"id":"5685245a-b502-4785-9f90-1d2ad6b0deee","arxiv_id":"2505.17872","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MoLA adapts a pre-trained short-horizon forecaster to multiple forecast steps via segment-specific mixtures of shared low-rank adapters, reporting modest mean-squared-error gains over the base models on most of eight benchmarks.","lead":"The paper claims a common decoding approach in time-series forecasting, predicting all future steps from one shared representation, carries an unavoidable accuracy penalty. It proposes a fix: pre-train for short horizons, then add per-horizon low-rank adapters with shared 'experts,' which helps on eight benchmarks, though mostly by narrow margins and with a theory that does not clearly establish the penalty.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1 does not analyze MT-F as trained: it fixes the decoder W and optimizes only R, so the claimed positive error independent of R is an artifact of the conditional least-squares setup, not a property of the co-trained MT-F objective.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: Theorem 3.1 computes the residual of the conditional least-squares problem with the decoder matrix fixed, whereas MT-F as trained also optimizes the decoder. This is not a minor wording issue; it is the entire basis for the paper's theoretical motivation. Once \\bar W is allowed to vary, the projection argument collapses and the lower bound can be driven to zero on a single sample, so the claimed independence from the encoder representation does not hold for the actual co-trained objective. The multivariate mismatch between the univariate proof and the LD-dimensional embeddings further weakens the transfer of the theorem to the evaluated models. The empirical results still provide some evidence that horizon-specific LoRA adaptation can improve forecasting accuracy over a shared-decoder baseline, and the paper includes ablations and sensitivity analyses that would be useful if the theoretical framing were repaired. Because the reader's conditional verdict already reflects the need for substantive revision of the central claim, I do not see grounds to move the verdict to reject or accept; the appropriate disposition remains conditional pending a corrected theorem and tempered claims.","tokens_in":28832,"tokens_out":6278,"duration_ms":58174,"concrete_test":"Re-derive Eq. (2) from the joint MT-F objective min_{\\bar W, \\bar R} \\|Y - \\bar W \\bar R\\|^2 instead of fixing \\bar W. For T=2, L=1, D=1, Y=(1,2), exhibit \\bar W=(1,2)^\\top, \\bar R=1, giving loss 0, directly contradicting the claimed strictly positive lower bound when T > L+1. If the proof cannot be extended to include \\bar W, the theorem should be restated as a property of a frozen decoder, and the abstract's claim of unavoidable errors with optimal representations should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that MT-F has an expressiveness bottleneck: when T > L+1, a strictly positive minimum error is unavoidable and independent of the encoder representation R. The Appendix C proof establishes only a weaker conditional statement. It fixes the decoder matrix \\bar W, solves \\hat{\\bar R} = (\\bar W^\\top \\bar W)^{-1} \\bar W^\\top Y, and reports the residual as the projection of Y onto the null space of \\bar W. But MT-F as trained co-optimizes \\bar W and the encoder; the theorem never optimizes \\bar W. This distinction is decisive. For a single label sequence with T=2, L=1, D=1 and Y=(1,2), choose \\bar W=(1,2)^\\top and \\bar R=1; the training loss is zero, so no positive lower bound exists. Across a dataset, the optimal error is min_W \\|Y - W R(X)\\|^2, which depends on the representation R(X); a degenerate encoder (e.g., R \\equiv 0) cannot be rescued by W, while a richer encoder can. The claimed invariance to R is therefore an artifact of optimizing only R while freezing W. The theorem also silently treats Y as a T-vector even though the paper defines Y \\in \\mathbb{R}^{T\\times D}; the rank bound L+1 does not transfer to the multivariate representations actually evaluated in the experiments. The expressiveness bottleneck, as stated, is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the standard multi-task forecasting (MT-F) paradigm, in which a single shared representation R is mapped to all forecast steps by a linear decoder with T outputs, suffers from an \"expressiveness bottleneck\": when the horizon exceeds the representation dimension plus one (T > L+1), a strictly positive minimum error is unavoidable and is independent of the encoder representation (Theorem 3.1). To avoid this bottleneck, the authors propose a two-stage framework: pre-train a model for S-step prediction (S ≤ L+1) and then adapt it to individual forecast segments using step-specific parameter-efficient modules. They instantiate this with Mixture-of-LoRA (MoLA), which shares P LoRA experts across K forecast segments through learnable segment-specific mixing weights δ_k. Experiments on ETT, ECL, Traffic, Weather, and PEMS benchmarks, using iTransformer and several backbones, show consistent improvements over the MT-F baseline and over state-of-the-art methods in most settings. The appendices include a proof of the bottleneck theorem, a variance-reduction argument for MoLA, complexity analysis, and extensive ablation results.","tokens_in":29102,"tokens_out":4025,"duration_ms":25575,"significance":"The paper sits at a timely intersection of parameter-efficient fine-tuning and time-series forecasting. If the expressiveness-bottleneck theorem were valid, it would provide a principled justification for replacing direct multi-step linear decoding with horizon-specific adaptation, which would be a meaningful conceptual contribution for the long-horizon forecasting community. The framework itself is simple, general, and empirically effective: MoLA consistently improves over the MT-F baseline across multiple backbones (iTransformer, Autoformer, Informer, Transformer) and datasets, with the full results table reporting 33/45 best MSE and 36/45 best MAE. The paper ships an anonymous code repository, reports ablations for K, r, η, P, layer-wise fine-tuning, and alternative PEFT modules (Adapter, IA3), and includes a parameter-count analysis showing that MoLA adds only a small fraction of parameters. These strengths make the empirical contribution credible and reproducible.","major_comments":[{"comment":"The bottleneck theorem does not analyze the MT-F model as trained. The proof fixes the decoder parameter W̄ and solves for the least-squares representation R̂ = (W̄ᵀW̄)⁻¹W̄ᵀY, reporting the residual in the null space of W̄. But in MT-F training, W̄ is co-optimized with the encoder representation R(X). For a single sequence with Y=(1,2)ᵀ, L=1, T=2, the decoder W̄=(1,2)ᵀ and R=1 give zero loss, so the claimed strictly positive lower bound when T > L+1 does not hold for the co-optimized objective. More generally, the optimal error min_{W,R} ‖Y − WR‖² depends on the representation: a degenerate encoder R=0 causes large error that no W can fix, while a rich encoder can achieve small error. The claimed independence from R is an artifact of the conditional least-squares setup and must be removed or substantially weakened.","section":"Theorem 3.1 and Appendix C (Proof of Theorem C.1)"},{"comment":"There is a dimensional mismatch between the theorem and the experiments. The theorem states Y ∈ R^{T×D}, R ∈ R^{L×D}, and the decoder parameter W̄ ∈ R^{T×(L+1)}, whose rank is at most min(T, L+1). The bound T > L+1 treats the representation as effectively univariate (dimension L). In the experiments, however, the encoder outputs a multivariate representation of dimension L times D (or hidden dimension d_m per variate in iTransformer), so the rank-deficient condition T > L+1 need not bind at all. As a result, the theorem's strictly positive error premise does not apply to the evaluated models. The authors should restate the theorem with the decoder as a general linear map from R^{L×D} to R^{T×D} (or make explicit the univariate assumption) and check whether the bottleneck condition can be satisfied under realistic hidden dimensions.","section":"Theorem 3.1, notation and rank bound"},{"comment":"The variance-reduction theorem is circular. It assumes ΔCov(L_t, L_s) = Cov_MTF(L_t, L_s) − Cov_MoLA(L_t, L_s) ≥ 0 and then concludes Var(L_MoLA) ≤ Var(L_MTF), which is exactly the assumed inequality restated. The text explains that modularized fine-tuning introduces horizon-specific parameters and therefore reduces parameter sharing, but no mechanism is given that guarantees the covariance difference is nonnegative, and in fact adding more parameters could in principle increase variance. To be non-circular, the authors would need to bound the covariance change under the actual MoLA update rule, or explicitly label the result as an empirically observed property rather than a theorem.","section":"Appendix C, Theorem C.2"},{"comment":"The use of Theorem 3.1 to justify segment sizes S ≤ L+1 is questionable. Even if the theorem were accepted for a fixed W̄, the bound L+1 refers to the representation dimension of the linear decoder input, but the paper does not specify how L is measured in the MoLA experiments: the foundation model's intermediate representation dimension is not the lookback length L in general. In iTransformer, for instance, the token embedding dimension is d_m, not the historical window length. The condition S ≤ L+1 should be checked against the actual representation dimension of the pre-trained backbone, otherwise the claim that MoLA bypasses the bottleneck is not supported by the theorem.","section":"Section 3.3 and Algorithm 1"}],"minor_comments":[{"comment":"The paper uses L both for the historical window length and for the representation dimension in Theorem 3.1, and the decoder is written as W ∈ R^{T×L} multiplying R ∈ R^{L×D}, which is dimensionally inconsistent (a T×L matrix times an L×D matrix gives T×D, but the text writes W_t R with W_t apparently a row). Please clarify the exact dimensions of R, W, and the bias, and use distinct symbols for window length and representation dimension.","section":"Equation (1) and notation throughout"},{"comment":"The statement that the adaptation stage uses a single-output decoder and thereby eliminates the error term inherits the issues of Theorem 3.1. The empirical motivation is clear, but the sentence should be softened to say the bottleneck bound does not apply, or should defer to the corrected theorem.","section":"Section 3.2, adaptation description"},{"comment":"The sensitivity discussion would benefit from noting that the Weather dataset shows a large discontinuous improvement at K=8 (MSE 0.173 vs 0.200–0.202 for other K values) at T=96, which may reflect lucky validation selection or a hyperparameter interaction rather than a smooth trend; a short remark would prevent over-interpretation.","section":"Table 4 and Figure 5"},{"comment":"The parameter-count ratio formula appears to double-count or miscount the per-segment weight parameters: the term P×K for the mixing weights has dimensions that do not match the other terms. Please check the derivation and clarify what is being counted.","section":"Appendix B, Equation (9)"},{"comment":"Several cells in Table 6 are bolded for non-MoLA entries (e.g., FreTS on Weather, DLinear on ETTm1 336) without discussion. Adding a sentence acknowledging that MoLA is not universally best would strengthen the empirical claims.","section":"Full result tables"},{"comment":"Reference [20] duplicates [19], and the reference list contains several formatting issues; please deduplicate and harmonize the bibliography style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is solid and the proposed framework is plausible, but the paper's central theoretical claim is currently not established. The theorem's conditional least-squares analysis fixes the decoder and optimizes only the representation, which is not the MT-F training objective; a co-optimization counterexample shows the claimed strictly positive error is not a property of the MT-F objective. In addition, the theorem's rank-bound premise does not match the multivariate hidden dimensions used in the experiments. The variance-reduction theorem in Appendix C is a restatement of its assumption rather than a proof. If the authors can either establish the bottleneck for the co-trained objective under the actual dimensions, or reframe the contribution as an empirical study of horizon-specific fine-tuning with the theorem as a heuristic motivation, the paper would be publishable. Given the strength of the empirical results, I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core of this paper is worth taking seriously. The MoLA architecture—segment-specific normalized mixing of shared LoRA experts—is new, the code is available, and the experiments are thorough: it improves four backbones on most benchmarks, with typical gains of 1–5% MSE and up to 13% on PEMS08. The sensitivity analysis on K and the layer-wise fine-tuning study are honest and useful. If you work on long-horizon forecasting, the method is a reasonable plug-in to try, though not a dramatic leap over standard fine-tuning.\n\nThe soft spot is the theory, and it is load-bearing. Theorem 3.1 claims an 'unavoidable, encoder-independent' error for MT-F, but the proof fixes the decoder W and optimizes only R. As trained, MT-F co-optimizes W and the encoder; the residual term in equation (2) is the projection onto the null space of a fixed W, and W can rotate to reduce or eliminate it. The rank bound L+1 also does not transfer to the multivariate representations (dimension LD) actually evaluated, where LD can exceed T. So the central claim that MT-F has an expressiveness bottleneck, as stated, is not established. The separate variance-reduction theorem (C.2) is circular—it assumes the covariance reduction it concludes. These are not minor typos; they are the theoretical justification for the two-stage design.\n\nThe abstract also overclaims: MoLA does not beat all SOTA on every dataset. On Weather, FreTS and TimesNet are better on average MSE; on ETTh1, FEDformer matches or beats it. The gains are real but more modest than 'outperforms state-of-the-art' suggests.\n\nMy bottom line: the method deserves a serious referee and the empirical work is publishable after revision, but the theory must be reframed—either state the theorem for a frozen decoder (which is true but trivial) or find a genuine bound for the co-trained objective. As it stands, the paper presents a useful empirical technique with an unsupported theoretical headline. I would send it to peer review, but with the expectation that the authors need to substantially revise the theoretical claims and temper the abstract.","headline":"The MoLA empirical recipe is plausible and mostly works, but the expressiveness-bottleneck theorem is not established as stated.","tokens_in":29760,"tokens_out":1484,"would_cite":true,"duration_ms":13709,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard multi-step forecasting reads every horizon from one shared representation; this paper proves that construction carries a strictly positive minimum error when the horizon exceeds the representation dimension, and introduces a…","keywords":["time-series forecasting","expressiveness bottleneck","multi-task forecasting","low-rank adaptation","mixture of experts","parameter-efficient fine-tuning","long-horizon forecasting"],"falsifier":"Train a standard MT-F model on a single label sequence whose $T$ future values are chosen so each row lies in $\\mathbb{R}^{L+1}$, jointly optimizing encoder and decoder weights; if the training residual can be driven to zero while $T > L+1$, the claimed representation-independent error floor does not appear in the actual training objective. A second check: with a multivariate embedding of dimension $L\\times D > T$, verify whether the theorem's predicted positive minimum error is observable at all.","tokens_in":28483,"feed_emoji":"📈","tokens_out":6131,"duration_ms":53439,"temperature":0.7,"pith_summary":"The paper claims that the dominant multi-task forecasting (MT-F) design—one encoder representation $R$ read out by a linear layer into all $T$ forecast steps—has an expressiveness bottleneck: when $T > L+1$, the minimum attainable error is strictly positive and independent of how good the encoder is. If true, this would mean every shared-representation multi-step decoder is irreducibly lossy, so gains must come from the decoding side. To bypass the bottleneck, the paper proposes pre-training a one-step foundation model and then adapting it with low-rank (LoRA) modules that are specific to each forecast horizon. The Mixture-of-LoRA (MoLA) variant segments the horizon and shares LoRA experts across segments with learned weights, preserving parameter sharing while removing the shared-representation constraint. On standard benchmarks, MoLA applied to iTransformer improves MSE/MAE over the base model and over state-of-the-art baselines on most settings.","feed_headline":"One shared representation caps multi-step forecast accuracy","feed_subtitle":"Adapting each forecast horizon with separate LoRA modules beats standard multi-task forecasting in tests.","key_machinery":"The load-bearing object is the 'expressiveness bottleneck': the projection of the label sequence $Y$ onto the null space of the decoder weight matrix $\\bar{W}$, which cannot be reached by any decoder output and therefore contributes unavoidable squared error $\\|U_t^\\top Y\\|_2^2$ per lost direction. The paper's remedy is a two-stage adaptation: first train a foundation model with a single-step output (where the null space vanishes), then freeze it and inject LoRA matrices $B^{(p)}A^{(p)}$ into selected layers for each forecast segment, with per-segment mixture weights $\\Delta_k$ combining a shared pool of $P$ LoRA experts. This keeps the number of decoder outputs per segment at $S=T/K\\leq L+1$, so the bottleneck term disappears, while the shared experts capture inter-step correlations.","core_discovery":"The paper's central claim is Theorem 3.1: with a fixed linear decoder $\\bar{W}\\in\\mathbb{R}^{T\\times(L+1)}$, the least-squares forecast error is $\\|\\epsilon\\|_F^2 = \\sum_{t=\\mathrm{rank}(\\bar{W})+1}^{T}\\|U_t^\\top Y\\|_2^2$, where $\\{U_t\\}$ span the null space of $\\bar{W}$; this error is positive whenever $T>L+1$ and is independent of the representation $R$ the encoder provides. The authors therefore argue that sharing one representation across all forecast steps caps achievable accuracy, and that the remedy is to give each step (or segment of steps) its own adapted representation while freezing the pretrained backbone. They report that MoLA outperforms both the MT-F and autoregressive paradigms on the tested datasets and that the benefit grows with horizon length.","pith_inferences":["The theorem's 'unavoidable error' is derived by fixing $\\bar{W}$ and optimizing only the augmented representation $\\bar{R}$; under the actual training objective, which co-optimizes $\\bar{W}$, the residual can be rotated away for a single label sequence, so the strongest form of the claim ('independent of the representation') likely does not hold for jointly trained models.","The rank condition is stated for a univariate representation of length $L$, while the experiments use multivariate embeddings of dimension $L\\times D$ that usually exceed $T$; that suggests the empirical gains may come more from horizon-specific fine-tuning and regularization than from the theorem's bound.","A direct test of the mechanism: use a linear encoder so the representation has exactly $L$ features, vary $T$ across the $L+1$ threshold, and see whether the error floor predicted by the theorem appears only on the $T > L+1$ side.","If the bottleneck argument is what motivates horizon-specific adaptation, the same logic would apply to other readout schemes, such as probabilistic forecasters or non-linear decoders, whenever a single representation is shared across outputs."],"forward_implications":["If the bottleneck is real, MT-F models with $T > L+1$ have an accuracy floor that no encoder improvement can remove.","Pre-training for one-step prediction and adapting per segment avoids that floor while keeping the base model frozen, so long-horizon adaptation does not degrade short-horizon performance.","Segmenting the horizon and sharing LoRA experts through a mixture-of-experts layer keeps added parameters modest while exploiting correlations between nearby and distant steps.","On the reported benchmarks, MoLA improves the average MSE/MAE of iTransformer on all nine datasets and surpasses the state-of-the-art baselines on most horizons.","The framework also transfers to other transformer backbones (Autoformer, Informer, Transformer) and to other parameter-efficient modules (Adapter, IA3), so the decoding fix is not tied to LoRA alone."],"supporting_citations":[{"why":"Supplies the iTransformer backbone and the MT-F paradigm that MoLA adapts and compares against.","marker":"[33]"},{"why":"Introduces the LoRA low-rank adaptation technique that the paper's per-step and per-segment modules build on.","marker":"[21]"},{"why":"Exemplifies the MT-F linear-decoder approach that the theorem analyzes and serves as a baseline.","marker":"[50]"},{"why":"Provides the benchmark datasets, chronological split protocol, and baseline reproduction settings used in the experiments.","marker":"[42]"},{"why":"Represents the MT-F forecasting paradigm (Informer) that the paper argues suffers from the expressiveness bottleneck.","marker":"[26]"},{"why":"Supplies Autoformer as a baseline and as one of the backbones tested with MoLA in the generalization study.","marker":"[43]"},{"why":"Serves as a frequency-domain MLP baseline whose competitive results on Weather are discussed in the full results.","marker":"[49]"}],"fun_headline_variants":["Per-step LoRA adapters break forecast accuracy ceiling","MoLA partial parameter sharing sidesteps shared bottleneck","Split forecast horizons, each with own LoRA, lift accuracy","Per-horizon LoRA experts sidestep shared bottleneck","MoLA: partial sharing per step lifts forecast accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of the unavoidable error fixes the decoder weights and optimizes only the representation, while real training also optimizes the decoder weights; the bound that makes $T>L+1$ matter also assumes a univariate representation of length $L$, which the benchmarked multivariate models exceed.","fun_headline_variants_meta":{"raw":{"variants":["Per-step LoRA adapters break forecast accuracy ceiling","MoLA partial parameter sharing sidesteps shared bottleneck","Split forecast horizons, each with own LoRA, lift accuracy","Per-horizon LoRA experts sidestep shared bottleneck","MoLA: partial sharing per step lifts forecast accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000961,"raw_usage":{"total_tokens":4064,"prompt_tokens":889,"completion_tokens":3175,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":3095}},"tokens_in":505,"tokens_out":3175,"duration_ms":22714,"temperature":1.0,"reasoning_tokens":3095,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:42:05.731522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a standard MT-F model on a single label sequence whose $T$ future values are chosen so each row lies in $\\mathbb{R}^{L+1}$, jointly optimizing encoder and decoder weights; if the training residual can be driven to zero while $T > L+1$, the claimed representation-independent error floor does not appear in the actual training objective. A second check: with a multivariate embedding of dimension $L\\times D > T$, verify whether the theorem's predicted positive minimum error is observable at all.","supporting_citations":[{"cited_title":"itrans- former: Inverted transformers are effective for time series forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies the iTransformer backbone and the MT-F paradigm that MoLA adapts and compares against."},{"cited_title":"Timesnet: Temporal 2d-variation modeling for general time series analysis","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark datasets, chronological split protocol, and baseline reproduction settings used in the experiments."},{"cited_title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","cited_arxiv_id":null,"evidence_quote":"Represents the MT-F forecasting paradigm (Informer) that the paper argues suffers from the expressiveness bottleneck."},{"cited_title":"Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies Autoformer as a baseline and as one of the backbones tested with MoLA in the generalization study."},{"cited_title":"Frequency-domain mlps are more effective learners in time series forecasting","cited_arxiv_id":null,"evidence_quote":"Serves as a frequency-domain MLP baseline whose competitive results on Weather are discussed in the full results."}],"review_version":1}