{"id":"ab85c203-6d54-40b3-95eb-32eeb6199535","arxiv_id":"2505.02655","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SCFormer, a channel-wise Transformer with triangular or convolutional temporal constraints and a HiPPO cumulative history state, improves forecasting on several benchmarks but not on Traffic.","lead":"A time-series forecasting model called SCFormer combines causal triangular or convolutional linear layers with a HiPPO memory state that encodes all history before the look-back window. It reports lower error than strong baselines on several standard datasets, but it loses on Traffic and the claimed theoretical motivation is not backed by significance tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal-causality claim is not isolated: the paper's own upper/lower-triangle equivalence contradicts its causal principle, so the reported gains may be sparsity/parameter reduction rather than a causal constraint.","rationale":"The reader identified the temporal-causality principle as the weakest assumption; my stress-test reaches the same point but sharpens it. The paper's key novelty is the structured triangular/convolutional projection, and the only direct evidence for it is Table 3(a), which does not isolate the causal direction from parameter reduction. The paper's own claim that upper and lower triangular matrices are equivalent actively undermines the causal story: if direction does not matter, then 'temporal constraint' is merely a particular kind of sparsity, not a causality-preserving operation. This is a load-bearing concern because the abstract and Section 1 present temporal constraints as the reason the method works, and the headline gains over iTransformer are attributed to the full SCFormer design. I do not see a reason to change the reader's CONDITIONAL verdict: the empirical recipe is plausible, code is available, and most results are consistent, but the causal-constraint attribution should be verified or softened. The proposed lower-triangular experiment is a direct, inexpensive way to settle the issue.","tokens_in":15197,"tokens_out":8669,"duration_ms":104754,"concrete_test":"Replace every upper-triangular projection in Eqs. (4) and (7) with the corresponding lower-triangular (time-reversed) projection, keeping nonzero count, initialization, and all hyperparameters identical, and run the same ECL/Weather benchmarks. If the lower-triangular variant matches SCFormer-triangular within seed noise, the improvement is not caused by causal direction; if it degrades, re-run with a dense low-rank projection matched to the same parameter count to test whether the gain is sparsity-induced regularization. This single experiment separates the causality story from the parameter-count story.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 motivates the central architectural choice by asserting that if the i-th element's feature a_i is computed as sum_j w_ij x_j, then the terms with j>i are 'unreasonable' and cause overfitting (Eq. 3). This is the load-bearing justification for replacing dense projections with triangular matrices. The concern is that this justification is not actually tested. The model predicts a future horizon from an observed look-back window; no autoregressive generation is performed, so causal masking of within-window features is not required by the task. The paper itself states that 'an upper or lower triangular matrix does not affect temporal constraints' (Section 3.2), even though lower-triangular projections would let later timesteps influence earlier ones and are therefore anti-causal under the paper's own definition. The only controlled ablation for the constraint, Table 3(a), varies sparsity and parameter count simultaneously (triangular cuts parameters roughly 50%, convolution roughly 90% per Section 4.3), so the small ETT gains are equally consistent with regularization or capacity reduction. No unconstrained-but-parameter-matched control is reported, and no constraint ablation is given on the datasets driving the headline improvements (ECL, Exchange, Weather). Thus the empirical architecture may be sound, but the stated principle that temporal causality produces the gains is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCFormer, a multivariate time series forecasting architecture that combines a channel-wise Transformer with two modifications: (i) a HiPPO-based cumulative historical state that encodes the full history before the look-back window, and (ii) structured linear transformations, implemented either as triangular matrices or 1D convolutions, intended to enforce temporal causality in query, key, value, and feed-forward projections. The authors report improvements over nine baselines on ETT, PEMS, Solar-Energy, ECL, Exchange, Weather, and Traffic, with the largest gains against iTransformer, and they claim these gains stem from the temporal constraints and the HiPPO state. Ablations and analyses of HiPPO variants, look-back length, and attention patterns are provided.","tokens_in":15412,"tokens_out":8700,"duration_ms":102105,"significance":"If the claims were fully supported, the paper would offer a simple and parameter-efficient way to inject long-range history into channel-wise Transformers, and the public code release is a strength. However, the evidence in the manuscript does not support the central attribution of the gains to the structured temporal constraints. In particular, Table 4 shows that on ECL, Weather, and Traffic, SCFormer-triangular achieves exactly the same MSE/MAE as the '+HiPPO' variant of the iTransformer-like baseline, indicating that the triangular constraint contributes nothing on those datasets. The paper's own ablations on ETT show only tiny, sometimes negative differences. The useful result appears to be the HiPPO cumulative-state injection, which is itself borrowed from prior work; the claimed novelty of structural temporal constraints is not experimentally established. The comparison is also confounded because SCFormer receives more input information (the cumulative history) than the baselines, which use only a 96-step look-back.","major_comments":[{"comment":"On ECL, Weather, and Traffic, SCFormer-triangular reports exactly the same MSE and MAE as the '+HiPPO' row of the Transformer baseline (e.g., ECL average 0.156/0.254, Weather 0.235/0.271, Traffic 0.509/0.359). Since the Transformer row appears to be the channel-wise iTransformer used elsewhere in the paper, this means the triangular temporal constraint yields no measurable benefit on these three datasets. The paper's headline improvements over iTransformer (12.3% on ECL, 8.9% on Weather) are therefore entirely attributable to the HiPPO state, not to the structured matrices. The authors must either explain this exact equality, provide a direct dense-parameter-matched control on all datasets, or substantially revise the claim that temporal constraints improve forecasting.","section":"§4.5, Table 4"},{"comment":"The temporal-causality justification is internally inconsistent and not tested in isolation. The paper states that 'an upper or lower triangular matrix does not affect temporal constraints' (Section 3.2), but under its own definition in Eq. (3), a lower-triangular matrix allows later timesteps to influence earlier ones; the two orientations are not equivalent with respect to the stated principle. More importantly, Table 3(a) compares SCFormer to Transformer-HiPPO on ETT only, varying sparsity and parameter count simultaneously, with differences that are tiny and sometimes negative (e.g., ETTm1 horizon 96: 0.318 vs 0.315). No constraint-only ablation with matched parameter count is reported on the datasets driving the headline results. The assertion in Section 4.4 that temporal constraints 'help mitigate overfitting' is not supported by the evidence presented.","section":"§3.2, Eq. (3); §4.4, Table 3(a)"},{"comment":"The experimental comparison is confounded by the cumulative historical state. Baselines are evaluated with a fixed look-back of 96, while SCFormer additionally receives the HiPPO state computed from the entire history. The improvement over iTransformer could therefore come from the extra input information rather than from the model architecture. Table 4 strongly suggests this is the case, as iTransformer+HiPPO matches SCFormer on the headline datasets. A controlled comparison should include baselines augmented with the same cumulative-state input, or an SCFormer variant without HiPPO evaluated under the same conditions. Without such a control, the claim that SCFormer 'significantly outperforms mainstream baselines' is not a fair attribution of architectural benefit.","section":"§3.1, Eq. (2); §4.2, Table 2"}],"minor_comments":[{"comment":"No error bars, confidence intervals, or significance tests are reported for any method, despite claims of 'significant' improvement. For the small differences in Table 3(a), multiple random seeds with standard deviations are necessary to support the conclusions.","section":"§4.3, Table 2"},{"comment":"Baseline results are taken from their original papers rather than rerun under a unified protocol; this should be disclosed alongside every comparison and ideally accompanied by a reproducibility statement for the baselines.","section":"§4.2"},{"comment":"The constraint is written as 'A_ij, B_ij, C_ij = 0 if i > j', but the value matrix is denoted E, not C; this notation error makes the equation ambiguous.","section":"§3.2, Eq. (4)"},{"comment":"The claim that 'at most ceil((d-k)/(k-1)) + 1 layers of convolution' generate a complete upper triangular matrix needs a proof or a citation; as stated, it is not obvious and is not derived in the text.","section":"§3.3, Eq. (10)"},{"comment":"The model labeled 'Transformer' in Table 4 is not defined in the text; if it is intended to be iTransformer (as the numbers suggest), this should be stated explicitly, and the caption should describe how the 'channel-wise strategy' is applied to Reformer, Informer, Flowformer, and Flashformer.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The exact numerical equality between SCFormer-triangular and the '+HiPPO' baseline in Table 4 is striking and is either a reporting artifact or evidence that the structured temporal constraint has no effect on the largest datasets. The authors should be asked to clarify this before any further consideration. The paper also does not rerun any baselines, so all cross-model comparisons are on unequal footing. The HiPPO component appears to carry the entire empirical improvement, which is incremental given the existing HiPPO literature; the structured-transformer contribution needs either much stronger evidence or removal from the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I'd want you to know before you read SCFormer: the empirical recipe is more credible than the story the authors tell about it. The architecture — HiPPO cumulative state feeding a channel-wise Transformer whose linear projections are triangular or 1D-convolutional — is new as a combination, and the reported gains (12% MSE on ECL, 17% on Exchange, 9% on Weather over iTransformer) are in a believable range. Code is public, ablations are thorough, and the HiPPO boost replicates across Reformer, Informer, Flowformer, and Flashformer. That is real work.\n\nThe soft spot is the temporal-causality justification. The paper claims unconstrained linear transforms are wrong because future elements shouldn't influence past ones. But the model reads a look-back window and outputs a future horizon; nothing in the task requires causal masking inside the window. The paper even says 'an upper or lower triangular matrix does not affect temporal constraints,' which contradicts its own definition: lower-triangular would let j>i influence a_i. So the stated principle is not just unproven; it's incoherent. The only ablation for the constraint (Table 3a) is on ETT, and it changes sparsity and parameter count at the same time, so the small gains could be regularization or capacity reduction. There is no parameter-matched unconstrained control, and no constraint ablation on ECL, Exchange, or Weather. The authors should either supply that control or drop the causal framing.\n\nOther weaknesses are more standard: baselines are quoted from their papers rather than rerun; no error bars or significance tests; the model loses on Traffic. None of these are disqualifying by themselves.\n\nWho benefits: practitioners who want a parameter-efficient channel-wise transformer with strong baseline numbers, and researchers working on state-space memory in forecasting. The flawed causal story doesn't invalidate the empirical contribution, but it needs to be fixed or removed before I'd trust the interpretation.\n\nI'd send it to peer review. Ask for a parameter-matched ablation of the constraint, error bars on the headline datasets, and a rewrite of Section 3.2. If they do that, this is a solid paper.","headline":"Worth a serious look as an empirical architecture, but the temporal-causality story is the weakest part and should not survive peer review as stated.","tokens_in":15960,"tokens_out":3529,"would_cite":false,"duration_ms":42966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that forcing all linear layers in a channel-wise Transformer to respect temporal order and feeding in a HiPPO-compressed cumulative historical state reduces multivariate forecasting error by up to 16.9% over a strong…","keywords":["multivariate time series forecasting","channel-wise Transformer","temporal causality","structured linear transformation","triangular matrix","1D convolution","HiPPO","cumulative historical state"],"falsifier":"Run the same SCFormer architecture with the triangular or convolutional constraints removed but the number of trainable parameters held fixed, for example by using dense matrices with matched rank; if the unconstrained model matches or beats SCFormer on ECL, Weather, and Exchange, then the temporal-constraint mechanism is not what drives the reported gains.","tokens_in":14965,"feed_emoji":"📉","tokens_out":6434,"duration_ms":73976,"temperature":0.7,"pith_summary":"SCFormer sets out to establish that multivariate time series forecasting by channel-wise Transformers has two fixable defects: unconstrained linear transformations let future values influence past features, and the model sees only the look-back window, ignoring history before it. The paper proposes forcing every linear map in the Transformer to respect temporal order, via triangular matrices or 1D convolutions, and feeding the model a HiPPO-compressed cumulative historical state alongside the look-back. Reported results show the triangular variant reduces average MSE by 12.3% on ECL, 16.9% on Exchange, and 8.9% on Weather relative to the main channel-wise baseline, while roughly halving parameters. A reader would care because both changes are simple and model-agnostic: if true, causality and long memory can be added to a transformer without redesigning attention.","feed_headline":"Triangular layers and HiPPO memory cut forecast error 17%","feed_subtitle":"Restricting transformer maps to time order and adding a compressed history state cuts error while halving parameters.","key_machinery":"Structured temporal-constraint matrices and HiPPO memory. The paper forces every linear map in the channel-wise Transformer to be upper triangular (or a 1D convolution, shown to be equivalent to a triangular Toeplitz matrix), so each time step's feature is built only from itself and earlier steps. HiPPO, High-order Polynomial Projection Operators, compresses the variable-length history before the look-back window into a fixed-size coefficient vector by projection onto orthogonal polynomial bases, updating it recursively; this state plays the role of the transition matrix in a Markov view, while the channel-wise Transformer acts as the emission model.","core_discovery":"The paper's central claim is that two additions to channel-wise attention improve multivariate time series forecasting: all linear transformations, including query, key, value, and feed-forward matrices, must be structured so that later time steps cannot influence earlier features, and the model should maintain a HiPPO projection of the entire history before the look-back window as a cumulative state, treating forecasting as emission from that memory rather than a Markov transition from a fixed window. The authors report that the triangular variant lowers average MSE by 12.3% on ECL, 16.9% on Exchange, and 8.9% on Weather compared with the strongest channel-wise baseline, and that replacing dense matrices with triangular or convolutional structure cuts parameters by about 50% or 90%, respectively. Ablations removing either component degrade performance, and attaching the HiPPO state to several other transformer variants improves most benchmarks.","pith_inferences":["The temporal-constraint principle is likely acting partly as an inductive bias or regularizer rather than a strict causal necessity, since a model can still read the full look-back window; a fair test would match parameter counts between constrained and unconstrained variants.","Because the HiPPO state is separate from the look-back, SCFormer could be combined with channel-independent or patch-based backbones, not only channel-wise attention, to test whether the memory mechanism transfers beyond attention over channels.","The paper's framing of forecasting as emission from a transition matrix invites a direct comparison with state-space sequence models, where the HiPPO update is already a standard memory primitive; such hybrids are a natural next step.","The attention-score and future-correlation analysis implies channel attention could be evaluated by how well it anticipates true inter-channel correlation in the forecast horizon, which suggests a diagnostic or auxiliary loss not explored in the paper."],"forward_implications":["Channel-wise attention can be made causally safe by structuring only the linear maps, eliminating future-to-past leakage in Q/K/V and feed-forward layers without changing the attention score formula.","The HiPPO cumulative state is portable: the paper's Table 4 shows adding it to Reformer, Informer, Flowformer, and Flashformer improves most benchmarks, so other transformer families can adopt the memory module independently.","Parameter efficiency follows structurally: triangular weight matrices contain about half the degrees of freedom of dense matrices and 1D convolutions roughly a tenth, so the accuracy gains come with a smaller model.","Look-back and cumulative history are complementary, not redundant: ablations that remove either component hurt accuracy, and lengthening the look-back still helps even when the cumulative state is present.","The Markov-to-emission reformulation suggests long-horizon forecasting can be treated as reading out a latent state of the whole series rather than extrapolating a fixed window."],"supporting_citations":[{"why":"Supplies the HiPPO projection operator that compresses cumulative history into a fixed-size state, the paper's central memory mechanism.","marker":"[6]"},{"why":"Provides the channel-wise Transformer architecture and hyperparameter configuration that SCFormer modifies, plus the strongest baseline it compares against.","marker":"[21]"},{"why":"Furnishes the linear-mapping baselines and the argument that over-long look-back windows blur global and short-term features.","marker":"[33]"},{"why":"One of the transformer baselines whose results define the comparison table and the claimed improvement.","marker":"[38]"},{"why":"Baseline using patched channel-independent input, representing the design alternative that SCFormer contrasts with channel-wise attention.","marker":"[22]"},{"why":"Prior memory-based transformer that approximates cumulative history with short- and long-term memory, which SCFormer positions HiPPO as a broader-horizon alternative.","marker":"[19]"},{"why":"Supplies reversible instance normalization, used to counter distribution shift in the preprocessing pipeline.","marker":"[11]"}],"fun_headline_variants":["Triangular layers and HiPPO memory slash forecast error","Halve parameters, boost forecast accuracy with triangular attention","Cumulative history state trims error in channel-wise transformer","Time-ordered transformer with long memory beats baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's results rest on the premise that letting future values influence how past time steps are represented is harmful enough to cause overfitting, rather than merely being a harmless modeling choice.","fun_headline_variants_meta":{"raw":{"variants":["Triangular layers and HiPPO memory slash forecast error","Halve parameters, boost forecast accuracy with triangular attention","Cumulative history state trims error in channel-wise transformer","Time-ordered transformer with long memory beats baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3903,"prompt_tokens":872,"completion_tokens":3031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":2965}},"tokens_in":488,"tokens_out":3031,"duration_ms":26597,"temperature":1.0,"reasoning_tokens":2965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:45:38.089318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SCFormer architecture with the triangular or convolutional constraints removed but the number of trainable parameters held fixed, for example by using dense matrices with matched rank; if the unconstrained model matches or beats SCFormer on ECL, Weather, and Exchange, then the temporal-constraint mechanism is not what drives the reported gains.","supporting_citations":[{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"One of the transformer baselines whose results define the comparison table and the claimed improvement."},{"cited_title":"In: The Eleventh International Conference on Learning Representations (2022)","cited_arxiv_id":null,"evidence_quote":"Baseline using patched channel-independent input, representing the design alternative that SCFormer contrasts with channel-wise attention."},{"cited_title":"Pattern Recognition Letters160, 26–33 (2022).https://doi.org/https: //doi.org/10.1016/j.patrec.2022.05.010,https://www.sciencedirect.com/ science/article/pii/S0167865522001623","cited_arxiv_id":null,"evidence_quote":"Prior memory-based transformer that approximates cumulative history with short- and long-term memory, which SCFormer positions HiPPO as a broader-horizon alternative."},{"cited_title":"In: In- ternational Conference on Learning Representations (2022),https://openreview","cited_arxiv_id":null,"evidence_quote":"Supplies reversible instance normalization, used to counter distribution shift in the preprocessing pipeline."}],"review_version":1}