{"id":"8348b3bb-60d8-4229-8b63-d2f5d1f89d7b","arxiv_id":"2509.23074","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A spectral-coherence-based score (SCP) and a Linear Utilization Ratio (LUR) are introduced to make time-series forecasting evaluation aware of intrinsic predictability, exposing predictability drift and architecture trade-offs.","lead":"This paper proposes a new score, SCP, that measures how hard a forecasting task is by looking at past-future coherence in the frequency domain, plus a diagnostic, LUR, that shows how much predictable signal a model uses. It matters because it reveals that simple linear models can win on easy tasks while complex models win on hard ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MSElb in Eq. (7) is not a lower bound for linear predictors: the Δ² term assumes a fixed DC gain of 1, but the optimal gain can make the true MSE lower than MSElb.","rationale":"The reader's weakest_assumption concerns the per-instance Welch coherence estimate and stationarity within windows. That is a legitimate statistical concern about the reliability of the estimate, but it is secondary. The more fundamental, load-bearing issue is that the proposed lower bound is not actually a lower bound for the class of predictors under consideration. The Δ² term in Eq. (7) is an arbitrary DC penalty (gain fixed to 1), not the minimal mean-mismatch contribution. A linear predictor can choose a different DC gain—or include a bias—to reduce or eliminate the mean error, while the AC residual remains the same. The counterexample with a shifted sine wave shows MSElb can exceed the achievable MSE by an arbitrarily large factor (here, 1 vs. 0). This invalidates the paper's central claim that MSElb is a lower bound and consequently undermines the interpretation of SCP as a predictability-aligned score. The empirical correlations in Table 1 may still hold numerically, but they do not validate the theoretical statement. Without a valid bound, the paper's claim to provide an 'irreducible error' baseline and the subsequent insights (predictability drift, architecture trade-offs) are not supported by the stated theory. The concern is concrete, easily tested, and directly falsifies the central equation. Therefore, the paper's central claim is not credible as written, and a major revision would be required to correct the bound or restrict the claim to a class where it actually holds.","tokens_in":13063,"tokens_out":23642,"duration_ms":190743,"concrete_test":"Implement Algorithm 1 exactly as specified (Welch with Hann window, n_win = floor(0.25N), 50% overlap, mean removal). Use a synthetic instance with N = 1024, x_t = 2 + sin(2πt/64), y_t = 0.5 x_t. Compute MSElb and SCP. Also compute the least-squares linear prediction error MSE_lin = min_{a,b} (1/N) Σ (y_t − a x_t − b)², which will be ≈ 0. If MSElb > MSE_lin (specifically, MSElb ≈ 1 while MSE_lin ≈ 0), the lower-bound claim in Eq. (7) is falsified.","verdict_should_be":"REJECT","load_bearing_attack":"The central mathematical claim in Section 4.1 is that MSElb = Δ² + Σ_f Ŝ_e(f) lower-bounds the MSE of any linear time-invariant predictor using x. This is not correct. After mean removal, the spectra capture only AC power; the mean mismatch is handled separately by Δ² = (mean(y) − mean(x))². But Δ² assumes the predictor has a DC gain of exactly 1. A linear predictor (even without an explicit bias term, if x has nonzero mean) can choose its DC gain to minimize the mean error, potentially reducing the total MSE below Δ² + Σ_f Ŝ_e(f). Concretely, take x_t = 2 + sin(2πt/64), y_t = 0.5 x_t = 1 + 0.5 sin(2πt/64). Then after mean removal, x' = sin(θ), y' = 0.5 sin(θ), so the coherence is 1 and the residual spectrum Σ Ŝ_e(f) ≈ 0. Δ² = (1 − 2)² = 1, so MSElb ≈ 1. But the linear predictor f(x) = 0.5x achieves MSE = 0, which is below the claimed lower bound. The bound therefore fails for the very class of predictors the paper evaluates, all of which include an affine/DC component. This is not a statistical estimation issue; it is an algebraic error in the bound. The SCP score and the interpretation that models cannot beat this 'fundamental limit' are built on this invalid bound.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a predictability-aligned evaluation framework for time-series forecasting. It defines a Spectral Coherence Predictability (SCP) score and a Linear Utilization Ratio (LUR), both based on Welch spectral estimates and squared coherence between history x and future y. The central theoretical claim is that MSElb = Δ² + Σ_f Ŝ_e(f) in Eq. (7) lower-bounds the MSE of any linear time-invariant predictor using x. The paper validates SCP on synthetic and real benchmarks, reports high correlation between MSElb and realized model MSE, and uses the framework to argue for 'predictability drift' and complementary strengths of linear vs. nonlinear architectures.","tokens_in":13390,"tokens_out":6130,"duration_ms":54985,"significance":"If the theoretical foundation were correct, the framework would be a valuable and computationally efficient diagnostic tool. The paper ships reproducible code, gives clear algorithms, and addresses an important problem: separating model error from irreducible data difficulty. However, the central lower-bound claim is false as stated. The error propagates into the definition of SCP, the interpretation of Fig. 1a, the correlation analyses in Table 1, and the 'fundamental limit' narrative. The other contributions (LUR, band-wise diagnostics) are interesting but rest on the same questionable spectral-coherence foundation, and the empirical validation is weakened by variance-induced correlation and a partly mechanical inverse relation between P and NMSE. I cannot recommend publication in the current form.","major_comments":[{"comment":"The claimed lower bound is not a lower bound for the stated class of predictors. After mean removal, the spectra capture AC power, while Δ²=(mean(y)-mean(x))² assumes the predictor has DC gain exactly 1. An affine/linear predictor with arbitrary DC gain can eliminate the mean error. Concrete counterexample: x_t = 2 + sin(2πt/64), y_t = 0.5 x_t = 1 + 0.5 sin(2πt/64). Then coherence is 1, residual spectral sum ≈0, Δ²=1, so MSElb≈1, but the linear predictor f(x)=0.5x has MSE=0. Since the models evaluated in the paper (DLinear, iTransformer, etc.) all include affine/bias components, the bound does not apply to them. This invalidates the theoretical interpretation in Section 4.1 and the validation claims in Sections 5.1–5.2. The bound would only hold for predictors with DC gain exactly 1, a restriction not stated or used. The correct unconstrained linear lower bound would be Σ_f Ŝ_e(f) withou","section":"Section 4.1, Eq. (7)"},{"comment":"The reported Pearson correlations R between MSE and MSElb are inflated by shared dependence on target variance. Both quantities scale with Var(y): MSElb contains Σ_f Ŝ_yy(f)(1−γ²), and model MSE is also proportional to Var(y) whenever the model is not much better than predicting the mean. Thus high R can arise even if the bound has no predictive content beyond variance. The paper should report partial correlations controlling for Var(y), or correlations of normalized errors (NMSE vs. MSElb/Var(y)). Without such a control, 'strongly correlates' in Section 5.2 does not substantiate the claimed alignment.","section":"Section 5.2, Table 1 and Fig. 2"},{"comment":"The toy study in Section C.1 validates the bound only for a single least-squares linear forecaster on one synthetic multiband Gaussian process, with 3 trials. This is insufficient to establish that MSElb is a valid lower bound 'for any linear time-invariant predictor,' especially given the DC-gain issue in Eq. (7). The experiment should include nonlinear predictors, processes with mismatched segment means, and varying Welch parameters. As written, Fig. 1a merely shows one Wiener filter's MSE cannot go below a quantity that is not actually a lower bound.","section":"Section 5.1, toy study"},{"comment":"The 'predictability drift' evidence is based on a single channel of ETTh1 at horizon N=96 (Fig. 3), with per-instance Welch estimates using n_win=⌊0.25N⌋=24 and 50% overlap. For short windows, the coherence estimate has high variance, and Appendix B explicitly acknowledges reliance on second-order stationarity within windows. A single-channel, single-dataset, high-variance estimate does not support the paper's claim of 'the first systematic evidence' of predictability drift. The authors should provide multi-dataset, multi-channel evidence with confidence intervals or statistical tests for drift.","section":"Section 5.3 and Appendix B/C.4"},{"comment":"The inverse relationship between P and NMSE is partly mechanical. Since MSElb ≤ MSE (if the bound were correct), NMSE = MSE/Var(y) ≥ MSElb/Var(y) = 1−P. Thus datasets with higher P necessarily have a higher lower bound on achievable NMSE. The observed inverse trend is therefore not independent evidence that P 'aligns with realized errors.' The text should treat Fig. 1b as a consistency check, not as validation of the metric.","section":"Section 5.2, Fig. 1b"}],"minor_comments":[{"comment":"P⋆ is defined as 1 − MSE⋆/Var(y). This is the theoretical R², but the paper should clarify that for finite samples and non-Gaussian processes it is an approximation; the current wording in Eq. (8) treats the estimator as automatically in [0,1].","section":"Section 3, Eq. (4)"},{"comment":"LUR is undefined when P_linear = 0. Some frequency bands may have zero estimated linear explainable power, especially with short windows. The algorithm should specify a default value or a guard condition.","section":"Algorithm 2, Eq. (12)"},{"comment":"The time-mixer citation in Section C.2 points to 'TimeXer (Wang et al., 2024)' rather than a TimeMixer reference; the reference list entry is for TimeXer, and the model name in Table 1 is TimeMixer. Please reconcile.","section":"Section 2 and Appendix C.2"},{"comment":"The Welch parameters are fixed as n_win=⌊0.25N⌋, 50% overlap. For small horizons (e.g., N=96), this yields only a few frequency bins; the paper should report sensitivity to Welch parameter choices, since the SCP/LUR values and the drift evidence depend on them.","section":"Appendix C.4"}],"recommendation":"reject","confidential_remarks":"The central mathematical error in Eq. (7) is decisive: the claimed lower bound is false for the class of predictors actually evaluated. The paper's framing and all downstream claims (SCP as a fundamental limit, validation in Fig. 1a/Table 1) depend on this bound. Even if the authors correct Eq. (7) by removing the Δ² term and re-running experiments, the correlational and drift evidence need substantial rework. I do not see a path to acceptance without a major rewrite of the theoretical core."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper has a decent idea—using spectral coherence to define a per-instance predictability score and a frequency-resolved utilization diagnostic—but the central mathematical claim is wrong as stated. Section 4.1 says MSElb = Δ² + ΣŜ_e(f) lower-bounds any linear time-invariant predictor's MSE. That only holds if the predictor's DC gain is fixed to 1. Once you allow a bias or an affine term—which all the evaluated models have—the predictor can set its DC gain to eliminate the mean mismatch, so the true MSE can be below MSElb. Simple counterexample: x_t = 2 + sin(θ), y_t = 0.5x_t. Coherence is 1, Δ²=1, MSElb≈1, but the predictor f(x)=0.5x achieves MSE=0. So the bound, and the P score built on it, is not a fundamental limit for the model class in question.\n\nWhat's actually new and useful: repackaging Wiener-filter spectral residual into an O(N log N) per-instance score, and the LUR band comparison, is a practical diagnostic toolkit. The 'predictability drift' observation could be interesting, but it's shown for a single channel with no systematic analysis or significance. The correlations in Table 1 look plausible but are vulnerable to shared dependence on target variance, and no uncertainty quantification is given. The paper itself admits the stationarity assumption in Appendix B; that's fine but it limits the interpretation.\n\nThis is not a desk-reject: the diagnostic framework has real potential, and the flaw is fixable—either restrict the predictor class to mean-matched predictors or recompute the bound with optimal DC gain. But the validation needs work: confidence intervals, partial correlations, multi-channel drift, and more datasets before the 'architecture trade-off' claim is taken seriously. I'd send it to review, with a clear note that Section 4.1 needs a major correction. The reproducibility statement is also just a promise; the authors should release actual code with a commit hash.","headline":"The SCP/LUR diagnostic idea is worth a look, but the paper's central 'lower bound' is not a lower bound for the predictors it evaluates, and the empirical validation is thinner than claimed.","tokens_in":13892,"tokens_out":3602,"would_cite":false,"duration_ms":33969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A spectral-coherence score separates a time series' intrinsic difficulty from a model's forecasting error, and a companion ratio reveals how much of the linear predictability each model actually exploits.","keywords":["time series forecasting","predictability","spectral coherence","model evaluation","linear utilization ratio","spectral coherence predictability","forecastability","frequency-domain analysis"],"falsifier":"Run SCP on a synthesized chirp or regime-switching process with a known linear forecaster and check whether the realized MSE ever falls below the claimed MSE_lb; a single violation on a stationary Gaussian process with known covariance would also falsify the bound's universality, as would a Monte Carlo where the Welch coherence estimate's confidence interval excludes the true coherence for a known spectrum.","tokens_in":12910,"feed_emoji":"📈","tokens_out":5930,"duration_ms":41545,"temperature":0.7,"pith_summary":"Standard forecasting metrics such as MSE cannot tell whether a model is failing because it is weak or because the data is inherently unpredictable. The paper proposes that the squared coherence between history and future, computed with Welch spectra, yields a lower bound on the error any linear predictor can achieve, giving a per-instance predictability score (SCP). A companion ratio (LUR) compares a model's prediction-target coherence to that linear limit, diagnosing under-use, saturation, or beyond-linear gains. The paper shows empirically that this framework predicts where models will struggle, reveals that predictability drifts sharply over time, and exposes complementary strengths of linear versus nonlinear architectures. If valid, it makes evaluation fairer and gives actionable guidance for model design.","feed_headline":"Coherence metric splits forecast error into data limit and model skill","feed_subtitle":"SCP quantifies intrinsic difficulty; LUR shows when models underuse, match, or beat the linear limit.","key_machinery":"Squared spectral coherence γ²_xy(f) between history and future, estimated by Welch's method, is the load-bearing object. Its complement (1 − γ²) scales the target spectrum to give the residual spectrum Ŝ_e(f) = Ŝ_yy(f)(1 − γ²_xy(f)); summing this over frequencies (plus a boundary mean-shift term Δ²) gives MSE_lb, the linear lower bound. SCP is the normalized inverse of that bound; LUR compares the prediction-target coherence against this same linear ceiling.","core_discovery":"The paper's central claim is that for a history x and future y, the quantity MSE_lb = Δ² + Σ_f Ŝ_yy(f)(1 − γ²_xy(f)) is a lower bound on the MSE of any linear time-invariant predictor using x, so the score P_xy = 1 − MSE_lb / dVar(y) measures the linearly predictable fraction of the target's variance. On jointly Gaussian wide-sense stationary segments this is a consistent estimator of the Bayes-risk predictability P*. The companion LUR = P_model / P_linear, with P_model and P_linear being power-weighted coherences between prediction and target and history and target, respectively, tells whether a model under-utilizes, saturates, or exceeds the linear information available. The paper asserts","pith_inferences":["If SCP is valid, benchmark leaderboards could be re-reported as a function of P, making claims like 'model A beats B' conditional on difficulty; this could change how model selection is done for deployed systems where the data regime is known.","Because SCP isolates linear predictability, a model with LUR > 1 signals real nonlinear or exogenous information capture; the framework could be used to audit whether a predictive model is genuinely learning dynamics or exploiting trivial correlations.","A testable extension: use SCP as a dynamic difficulty measure for active learning or online learning, re-estimating predictability on a rolling window to guide when to retrain or switch models.","The per-instance lower bound is only for linear time-invariant predictors; a natural extension would be a comparable bound for nonlinear classes (e.g., via conditional coherence or transfer entropy) to tighten the baseline."],"forward_implications":["A model's MSE can be decomposed into an intrinsic data-dependent floor (MSE_lb) plus the model's extra error; evaluations can be stratified by SCP to compare models only within comparable difficulty.","Predictability drift implies aggregate test-set statistics are misleading; reporting should be per-sample or per-band and conditioned on predictability.","Band-wise LUR reveals that linear models behave as broadband filters while transformers focus on high-energy low-frequency bands, explaining their comparative strengths.","SCP can guide curriculum learning and data sampling by difficulty, and LUR can inform regularizers; the paper suggests these as future uses."],"fun_headline_variants":["Coherence metric isolates data limit from model skill","Forecast error decomposed: data unpredictability vs model inefficiency","New metric shows forecast errors partly due to data's own noise","Study reveals predictability drift in time series tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework's load-bearing premise (acknowledged in Appendix B) is that a single short history–future pair, windowed with length ⌊0.25N⌋, gives a reliable squared-coherence estimate under local wide-sense stationarity; if real series are not stationary at that scale, SCP and LUR become noisy and MSE_lb may not be a valid lower bound.","fun_headline_variants_meta":{"raw":{"variants":["Coherence metric isolates data limit from model skill","Forecast error decomposed: data unpredictability vs model inefficiency","New metric shows forecast errors partly due to data's own noise","Study reveals predictability drift in time series tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2516,"prompt_tokens":780,"completion_tokens":1736,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1672}},"tokens_in":524,"tokens_out":1736,"duration_ms":9847,"temperature":1.0,"reasoning_tokens":1672,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T14:46:42.047080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SCP on a synthesized chirp or regime-switching process with a known linear forecaster and check whether the realized MSE ever falls below the claimed MSE_lb; a single violation on a stationary Gaussian process with known covariance would also falsify the bound's universality, as would a Monte Carlo where the Welch coherence estimate's confidence interval excludes the true coherence for a known spectrum.","supporting_citations":[],"review_version":1}