{"id":"fc65e4ed-51cc-450e-b75d-659c5213bcbd","arxiv_id":"2502.06037","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"On a benchmark where models train on Fourier components and test on their sums, patch-based Transformers and residual MLP architectures show the strongest compositional generalization, while most standard transformers and linear models fail.","lead":"These authors ask whether time series forecasting models can combine simple patterns learned during training to forecast more complex, never-seen combinations. They train 16 models on individual sine and cosine components of signals, then test on the original combined signals, and find that patch-based Transformers and residual MLP models generalize best.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark's addition operation cannot distinguish compositional reasoning from linear superposition; the paper's controls do not rule out linear additivity, so the central reasoning claim is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the benchmark defines compositional reasoning as forecasting a summed signal after training on its Fourier components, but addition can be solved by linear superposition. I agree that this construct-validity issue is the most central threat. The efficiency comparison (97% FLOPs, 86% parameters) and the architecture ranking may survive as empirical facts about spectral extrapolation, but the paper's title and headline 'reasoning' claim depend on the benchmark actually measuring composition of concepts rather than additivity. I do not think this concern overturns the paper entirely: the empirical pattern is consistent, and the winning models do generalize better on the proposed task. However, the current evidence does not establish that the mechanism is compositional reasoning. The proposed linear control and additivity probe would settle this directly. If the linear control matches the winning models, the paper should be reframed as a study of additive spectral extrapolation; if it does not, the reasoning interpretation gains support. Other issues raised by the reader—patch-length selection on test MAE, the absence of actual pretrained TSFMs, and the 'zero-shot' label—are real but secondary to this construct-validity concern. For these reasons I would keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":30132,"tokens_out":8691,"duration_ms":94360,"concrete_test":"Train a strictly linear model on the raw lag features under the identical §3.1 protocol and compare its OOD MAE and Top-k Basis Wins to T5/N-HiTS/N-BEATS/MLP. If the linear control matches their performance, the benchmark is solved by superposition and the reasoning interpretation is falsified. Additionally, for the winning models, compute f(s1+s2) versus f(s1)+f(s2) on held-out component pairs; if the normalized deviation is near zero, success is explained by additivity rather than compositional reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that T5, N-HiTS, N-BEATS, and MLP show compositional reasoning while most other architectures do not—rests on the assumption that success on the §3.1 benchmark requires composing concepts rather than exploiting additivity. The training set is the top-k Fourier components of y; the test set is the sum y (Eqs. 5–7). Because composition is addition, an approximately linear input–output map can pass the task by superposition: it has learned per-frequency extrapolation and sums the outputs. The paper's Top-2 Basis Win threshold (§3.4) does not separate these cases; it only excludes models that cannot beat a two-component reconstruction, which is a weak bar. The linear baselines DLinear/NLinear fail, but they are not controls for all linear mechanisms: they use moving-average decomposition and input normalization that can break additivity. N-HiTS and N-BEATS are additive basis-expansion models by design, so their success on an additive composition task is expected from their inductive bias, not evidence of emergent reasoning. The CKA analysis (§7.1) shows representational similarity but is correlational, is computed for one T5 variant, and does not show that the winning models' forecasts arise from composing separable component representations. Thus the ranking may be a robust statement about additive spectral extrapolation, but the 'reasoning' interpretation is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a benchmark for 'compositional reasoning' in time series forecasting: models are trained on the top-k Fourier components of a target signal (Eqs. 5–6) and evaluated zero-shot on the composed sum signal (Eq. 7). A new metric, Top-k Basis Win (Eq. 9), is proposed, with a threshold of k=2 used to identify models that show 'sparks' of compositional reasoning. The paper evaluates 16 deep learning and statistical forecasting models on one synthetic and five real-world datasets, plus two nonstationary synthetic benchmarks in the appendix, and reports that patch-based Transformers (PatchTST, T5) and residualized MLP-based models (MLP, NHITS, NBEATS) perform best, with NHITS/NBEATS being far more parameter- and FLOP-efficient. It also ablates design choices of a modular T5-based Transformer, finding patching to be the most beneficial tokenization method. The main claims are that these models exhibit compositional reasoning and that such reasoning does not simply follow from in-distribution forecasting performance.","tokens_in":30334,"tokens_out":7885,"duration_ms":83778,"significance":"If the operationalization is accepted, the paper would provide a novel and reusable benchmark for a largely unexplored capability of time series models, together with a broad, reproducible comparison and controlled ablations. Strengths include public code and datasets, evaluation across many architectures, statistical significance testing, and the explicit nonstationary extensions in the appendix. However, the significance is conditional: the central 'reasoning' interpretation is not supported by the current experimental design, and the headline ranking relies on test-selected patch lengths. The paper is best viewed at present as an empirical study of additive spectral composition, with the reasoning claim requiring additional controls.","major_comments":[{"comment":"The benchmark composes concepts by addition, so an approximately linear input–output map trained on individual basis functions can pass the test by superposition without any internal composition of concepts. The Top-k Basis Win threshold with k=2 only excludes forecasts that are worse than the dominant single-frequency reconstruction; it does not rule out linear additivity. The linear baselines DLinear and NLinear are not adequate controls because their moving-average decomposition and input normalization break simple additivity, and NHITS and NBEATS are additive basis-expansion architectures by design, so their success is expected from their inductive bias. The paper should either add explicit controls (for example, a linear model with Fourier features trained on the basis functions and tested on sums, or a nonlinear composition operation such as a product) or reframe the claims as being about additive compositional generalization rather than compositional reasoning.","section":"§3.1, Eqs. (5)–(7); §3.4, Eq. (9)"},{"comment":"The entries 'PatchTST (Best PL)' and 'T5 Model (Best PL)' appear to select the patch length per dataset from the test-set MAE tables. For example, in Table 9 the T5 Model OOD ECL value 0.239 corresponds to PL=96 while the Subseasonal value 1.714 corresponds to PL=128; in Table 8, PatchTST OOD Synthetic Sinusoid 10.696 is PL=128 while ETTm2 0.250 is PL=96. Selecting hyperparameters on the test set inflates the reported performance and undermines the central ranking claim. The paper should either report results for a single configuration chosen without test data, or report all patch lengths and base the ranking on a validation-based selection.","section":"Table 2 and §4"},{"comment":"The real-world benchmark is restricted to segments that pass the Augmented Dickey-Fuller test at α=0.001 and then to the 100 segments with the highest mean ACF. This selects the most stationary, most periodic subseries and excludes nonstationary behavior, which the paper acknowledges is left for future work. However, the abstract and Section 4 make broad claims about zero-shot OOD performance on 'all 6 datasets' and about outperforming statistical baselines on real-world data. The authors should report how this selection affects the conclusions, for example by including results on the full datasets or a sensitivity analysis over the ACF selection threshold.","section":"§3.2, Preprocessing"},{"comment":"The CKA analysis is correlational and is reported for a single architecture (T5 Model with patch length 128) on a single synthetic setting. It shows that embeddings of a model trained on both frequency and trend components are more similar to the ID ground-truth model than embeddings of models trained on one component, but it does not establish that the winning models' forecasts arise from composing separable component representations. This evidence should be presented as suggestive and clearly distinguished from the benchmark-based claims, which are already subject to the linear-superposition concern in my first major comment.","section":"§7.1, Table 1"}],"minor_comments":[{"comment":"There is a duplicated word in 'through through the addition of concepts' in the last paragraph of Section 2.","section":"Section 2"},{"comment":"'Casual' should be 'Causal' in the attention ablation entries.","section":"Table 4 and Figure 8"},{"comment":"The Top-k Basis Win definition should specify that L is MAE and clarify how the maximum in 'Top-kmax Basis Win' is computed and averaged across datasets; for noisy real signals, the maximum over k up to n is not a well-defined summary without this clarification.","section":"§3.4, Eq. (9)"},{"comment":"There is a typographical error in the standard deviation formatting: '(0.039) 0.039)' should be a single properly parenthesized value.","section":"Table 8, T5 Model row"},{"comment":"The abstract and Section 3.3 state that the paper examines design choices in '7 popular open-source TSFMs', but the controlled studies actually train a modular T5-based architecture with components inspired by those TSFMs; the seven TSFMs themselves are not trained. This should be rephrased to avoid overstating the scope.","section":"Abstract and §3.3"},{"comment":"The Friedman test is run at α=0.2, which is an unusually liberal significance level; the authors should justify this choice or also report results at conventional levels such as α=0.05. The figure captions should also state explicitly that the reported 'Difference' is ID minus OOD MAE.","section":"§3.4 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical contribution is real and reproducible, but the central 'reasoning' label is currently not supported by the experimental design, since the benchmark's composition operation is addition and the controls do not exclude linear superposition. The per-dataset best-patch-length selection in Table 2 is a more serious methodological issue than the authors' framing suggests and should be fixed through validation-based selection or by reporting all configurations. With a reframing toward additive compositional generalization and the addition of linear controls, the paper could be a solid contribution; in its present form, the overclaiming is too central to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper delivers a clean, reproducible spectral composition benchmark and a consistent architecture ranking: patch-based Transformers (T5, PatchTST) and residual MLP stacks (N-HiTS, N-BEATS) beat most other models on forecasting summed signals after training on their top-k Fourier components, and the MLPs do it with a fraction of the parameters. That is a real empirical result, and the code/data pipeline (NeuralForecast, GIFT-Eval) is a solid base for follow-ups. The Top-k Basis Win metric is a useful addition; it gives a threshold for ruling out models that can't beat a simple two-component reconstruction.\n\nWhere the paper overreaches is the word 'reasoning.' The composition operation is addition. A model that learns per-frequency extrapolation and sums its outputs will pass the task without any internal act of composition, and the linear baselines that fail (DLinear/NLinear) don't control for this: they use moving-average decomposition and input normalization that can break additivity. N-HiTS and N-BEATS are additive basis-expansion models by design, so their success on an additive task is, at least partly, their inductive bias doing the work. The CKA analysis is correlational and computed for one T5 variant; it shows representational alignment, not that the forecasts are produced by composing separable components. The paper itself never addresses this alternative explanation, even in the limitations section.\n\nThere are also smaller but real issues: the reported best patch lengths are selected from test-set tables (a validation split would be cleaner); the real-world benchmark keeps only the 100 stationary, highest-ACF segments, which narrows the claim; and the T5 model is called a 'foundation model' and the protocol 'zero-shot' despite training on the target series' components, which is standard transfer learning, not zero-shot in the TSFM sense.\n\nThe architecture ranking on this spectral extrapolation task is likely robust, and the benchmark is worth having. But the paper's framing as evidence of compositional reasoning in time series foundation models is not supported. With a relabeled scope—say, 'generalization to additive spectral compositions'—plus validation-based hyperparameter selection and a deeper control for linear additivity (e.g., a deliberately linear model with no decomposition, or an oracle that sums per-frequency forecasts), this could be a solid contribution.\n\nMy recommendation: send it to peer review. A serious referee can push on the reasoning interpretation and ask for the linear-superposition control. The empirical core is worth publishing, but not in its current framing.","headline":"Clean spectral-extrapolation benchmark and a robust architecture ranking, but the compositional-reasoning label outruns the evidence.","tokens_in":30964,"tokens_out":3542,"would_cite":true,"duration_ms":28815,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Time series models can be tested for compositional reasoning, and patch-based Transformers plus small residual MLP networks pass the test while most other architectures fail.","keywords":["compositional reasoning","time series foundation models","zero-shot forecasting","spectral decomposition","patch-based transformers","out-of-distribution generalization","representation similarity","model efficiency"],"falsifier":"Train a strictly linear forecaster (for example ridge regression on the context window) on the same top-$k$ Fourier component training set used for the deep models. If it attains Top-$k$ Basis Wins comparable to N-HiTS, N-BEATS, or the T5 model on the out-of-distribution test sets, the benchmark does not isolate compositional reasoning from linear additivity; a multiplication-based composition version of the benchmark would settle the question directly.","tokens_in":29874,"feed_emoji":"📈","tokens_out":11539,"duration_ms":106448,"temperature":0.7,"pith_summary":"The paper asks whether time series forecasting models succeed by memorizing training patterns or by reasoning about them, and it proposes a concrete test for one form of reasoning. It defines compositional reasoning as the ability to forecast a composite out-of-distribution signal after training only on that signal's Fourier components, then evaluates 16 deep learning models and two statistical baselines on synthetic and real-world data. The central finding is that patch-based Transformers and residual MLP architectures, notably the T5-based model, N-HiTS, N-BEATS, and a plain MLP, pass the test while most RNN, CNN, linear, and other Transformer variants do not. The efficient winners matter: N-HiTS and N-BEATS reach near-best accuracy with 97% fewer floating-point operations (FLOPs) and 86% fewer parameters than the best Transformer configuration. Component ablations show that input patching is the decisive design choice for Transformers, while larger model size does not improve reasoning.","feed_headline":"Patch transformers and small MLPs can reason about time series","feed_subtitle":"A spectral test shows models that forecast unseen sums of learned patterns, with the cheapest running at 3% of the best transformer's FLOPs.","key_machinery":"The load-bearing object is the spectral composition benchmark itself: the training set holds the top-$k$ discrete Fourier components of each target series (Eqs. 5–7), while the test set holds the summed signal, so the training and test distributions differ by construction and success requires composing what was learned. The Top-$k$ Basis Win metric (Eq. 9) supplies the evidence threshold — a model must beat a reconstruction from the top-$k$ Fourier components of the target to count as a reasoner. Auxiliary evidence comes from Centered Kernel Alignment on model embeddings, which tests whether composition appears in the latent space, and from critical-difference statistical tests that rank models and design choices across datasets. Controlled ablation over a modular T5-encoder Transformer isolates which components matter, with fixed-length input patching emerging as the one that unlocks reasoning.","core_discovery":"The authors claim that compositional reasoning is measurable in forecasting with a spectral benchmark: train a model on the top-$k$ Fourier basis functions of a time series and evaluate it zero-shot on the full summed signal. A model counts as a reasoner when, by their proposed Top-$k$ Basis Win metric, it matches or beats reconstruction from the top-$k$ basis at threshold $k \\geq 2$ — outperforming that simple benchmark rules out merely recalling single learned frequencies. On this test, the T5-based patch Transformer, N-HiTS, N-BEATS, and the plain MLP demonstrate compositional reasoning across stationary real-world datasets and on nonstationary trend-plus-sinusoid compositions, whereas iTransformer, Autoformer, TSMixer, LSTM, and most others do not. Centered Kernel Alignment of latent embeddings shows that a model trained jointly on trend and frequency components aligns more closely with a model trained on the composed ground-truth signal, which the authors take as evidence that composition occurs inside the representations. They further claim that reasoning does not scale with model size: small T5 variants rank best out-of-distribution, and among the tested components the tokenization method has the largest significant effect on Transformer reasoning performance.","pith_inferences":["A ridge-regression or other strictly linear control trained on the same Fourier components would sharpen the central claim: the paper's account predicts it fails the Top-$k$ threshold while the residual MLP models pass, but that exact control is not run.","Composing signals by a non-additive operation (multiplication, gating, thresholding) would separate genuine composition from linear additivity and is the most direct next benchmark.","The divergence between in-distribution and out-of-distribution rankings implies that model selection should evaluate OOD compositional tasks explicitly rather than trusting ID validation alone.","Since patching — preserving local temporal structure in the input tokens — is the decisive component, testing other local structure-preserving tokenizers such as wavelet packets is a natural follow-up."],"forward_implications":["Reasoning is not a property of whole model classes: among Transformers, patching-based models exhibit compositional reasoning while several other Transformer variants do not.","Efficient architectures suffice for reasoning: N-HiTS and N-BEATS match the best patch Transformer while using 97% fewer FLOPs and 86% fewer trainable parameters.","Zero-shot out-of-distribution forecasts from the reasoning-capable models can beat moving-average ARIMA on all six datasets and AutoETS on two, even though the statistical baselines were fitted on in-distribution data.","Model size and reasoning diverge: smaller T5 variants rank best on out-of-distribution tasks while larger variants win in-distribution tasks, so scaling alone is not a route to better reasoning.","Among design choices, tokenization has the clearest and largest effect — patching helps and binning hurts — while model size and projection layer show smaller significant differences and many other components (attention type, positional encoding, loss, scaler, context length, decomposition) do not change reasoning performance significantly."],"supporting_citations":[{"why":"Supplies the real-world datasets (electricity transformer temperature, electricity consumption, solar, climate, traffic) and the 48-step forecast-horizon protocol used for the OOD and ID evaluation.","marker":"[1]"},{"why":"Introduces N-HiTS, the hierarchical-interpolation MLP model that matches the best Transformer at 97% fewer FLOPs.","marker":"[8]"},{"why":"Introduces PatchTST, the patch-based Transformer whose success motivates the conclusion that input patching unlocks reasoning.","marker":"[37]"},{"why":"Introduces N-BEATS, the residual MLP architecture with basis expansion that ranks among the top generalizers and anchors the parameter-efficiency comparison.","marker":"[40]"},{"why":"Supplies the definition of compositional and implicit reasoning from the LLM literature that the benchmark adapts to forecasting.","marker":"[48]"},{"why":"Provides the multi-hop composition framework in language models that motivates the synthetic compositional task design.","marker":"[57]"},{"why":"Provides the critical-difference diagram methodology used for model ranking and for the statistical comparison of architecture components.","marker":"[14]"},{"why":"Provides the open-source T5-encoder backbone used in the modular ablation studies of architecture components.","marker":"[51]"},{"why":"Contributes the code used to generate the synthetic sinusoid datasets and supplies one of the compared open-source foundation models.","marker":"[21]"}],"fun_headline_variants":["Patch transformers and slim MLPs pass time series spectral reasoning test","MLPs nearly match transformers in time series reasoning, at 3% FLOPs","Spectral top-k test identifies which time series models reason compositionally","Small MLPs beat large LSTMs on compositional reasoning in zero-shot forecasting","Tiny MLPs: 97% fewer FLOPs, 86% fewer params, close reasoning to transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on the premise that forecasting a summed signal after training on its parts measures reasoning: because the composition is addition, an approximately linear model could pass by superposition, and a nonlinear model could fail for numerical rather than conceptual reasons.","fun_headline_variants_meta":{"raw":{"variants":["Patch transformers and slim MLPs pass time series spectral reasoning test","MLPs nearly match transformers in time series reasoning, at 3% FLOPs","Spectral top-k test identifies which time series models reason compositionally","Small MLPs beat large LSTMs on compositional reasoning in zero-shot forecasting","Tiny MLPs: 97% fewer FLOPs, 86% fewer params, close reasoning to transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001198,"raw_usage":{"total_tokens":4997,"prompt_tokens":1060,"completion_tokens":3937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":3832}},"tokens_in":676,"tokens_out":3937,"duration_ms":31366,"temperature":1.0,"reasoning_tokens":3832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:58:12.878339+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a strictly linear forecaster (for example ridge regression on the context window) on the same top-$k$ Fourier component training set used for the deep models. If it attains Top-$k$ Basis Wins comparable to N-HiTS, N-BEATS, or the T5 model on the out-of-distribution test sets, the benchmark does not isolate compositional reasoning from linear additivity; a multiplication-based composition version of the benchmark would settle the question directly.","supporting_citations":[{"cited_title":"Olivares, Boris N","cited_arxiv_id":null,"evidence_quote":"Introduces N-HiTS, the hierarchical-interpolation MLP model that matches the best Transformer at 97% fewer FLOPs."},{"cited_title":"Nguyen, and Phanwadee Sinthong an Jayant Kalagnanam2","cited_arxiv_id":null,"evidence_quote":"Introduces PatchTST, the patch-based Transformer whose success motivates the conclusion that input patching unlocks reasoning."},{"cited_title":"Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio","cited_arxiv_id":null,"evidence_quote":"Introduces N-BEATS, the residual MLP architecture with basis expansion that ranks among the top generalizers and anchors the parameter-efficiency comparison."},{"cited_title":"Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of compositional and implicit reasoning from the LLM literature that the benchmark adapts to forecasting."},{"cited_title":"Statistical comparisons of classifiers over multiple data sets","cited_arxiv_id":null,"evidence_quote":"Provides the critical-difference diagram methodology used for model ranking and for the statistical comparison of architecture components."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the open-source T5-encoder backbone used in the modular ablation studies of architecture components."},{"cited_title":"MOMENT: A family of open time-series foundation models","cited_arxiv_id":null,"evidence_quote":"Contributes the code used to generate the synthetic sinusoid datasets and supplies one of the compared open-source foundation models."}],"review_version":1}