{"id":"2398cfb4-0ed6-4ae5-aae0-3cd4d0392e19","arxiv_id":"2505.02011","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A CNN autoencoder that computes attention scores from variate correlations replaces self-attention, cutting resource use while improving multivariate time-series forecast accuracy.","lead":"CASA replaces the standard attention score computation with a CNN autoencoder that captures correlations between time-series variates. The authors report reduced memory use, faster inference, and first-place results on most benchmark metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8)'s element-wise softmax(Score)⊙V cannot reproduce the pairwise row-mixing of softmax(QK^T)V; CASA is a CNN gating mechanism, not an approximation of attention scores.","rationale":"The most load-bearing assumption is hidden in Eq. (8): CASA claims to approximate the attention score matrix, but its forward operation is element-wise multiplication with V. For any input Z∈R^{N×D}, the value V=f(Z) has shape N×D, so Score(Z) must also be N×D. Standard attention in Eq. (4) outputs Σ_j softmax(QK^T/√d)_{ij} V_j, which mixes information across all variates j into output row i. CASA outputs softmax(s_i)⊙v_i, a per-token gate. The CNN can make the gate depend on all rows, but that is a context-dependent scaling, not a weighted aggregation of other variates' values. Therefore the Proposition 1/2 motivation, which criticizes attention for not embedding cross-variate info in Q/K, is not addressed by this design: the cross-variate information never enters the output through value mixing. This is not a matter of numerical approximation; the two operations have different algebraic forms. The reported accuracy could still be genuine—CASA may be a strong gated CNN/MLP hybrid—but the paper's central claim that it approximates QK^T and improves cross-dimensional attention is unsupported. The reader's complexity concern is real but secondary; even if D^2 is negligible, the architecture would still not be an attention approximation. A concrete code check can settle this immediately.","tokens_in":17607,"tokens_out":7403,"duration_ms":83843,"concrete_test":"Inspect the released code or run a forward pass and record the shape of Score(Z) for a channel-wise input with N=862, D=512. If the tensor is (N,D) and the forward op is element-wise multiplication with V, compute the Jacobian ∂out_i/∂V_j for i≠j. In standard attention this is nonzero (A_{ij}); in CASA it is zero through the direct path, because out_i = softmax(s_i)⊙v_i. If zero, CASA performs no cross-variate value aggregation, contradicting the claim that it approximates QK^T. Alternatively, modify Eq. (8) to use true matrix multiplication with an N×N score map and check whether accuracy changes; if it does not, pairwise mixing is not the source of the reported gains.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that CASA approximates QK^T/sqrt(d_k) with a CNN autoencoder (Section 3.4). Under channel-wise tokenization, Z_i ∈ R^{N×D} and V_{i+1}=f(Z_i) ∈ R^{N×D}. Equation (8) defines Attention(Z_{i+1}) = softmax(Score(Z_i)) ⊛ V_{i+1}. Since ⊛ is element-wise, Score(Z_i) must also be in R^{N×D}; the stated score-network cost O(N k D^2) confirms this is a per-token feature map, not an N×N pairwise matrix. Standard self-attention in Eq. (4) computes softmax(QK^T/√d_k) V, so each output row is a weighted sum of all value rows. CASA's output row i is v_i times a scalar gate: there is no cross-variate value mixing. The convolution over variates lets the gate for row i depend on other rows, but the model never aggregates information from other variates into the output of variate i. Thus the Proposition-motivated story that CASA repairs the missing cross-variate information in query/key embeddings does not follow; what is implemented is a CNN-driven channel-gating mechanism, not an approximation of attention scores. This does not by itself refute the empirical accuracy results, but it removes the theoretical basis for 'Score Attention' and for the claimed cross-dimensional interaction mechanism, and it changes how the efficiency and ablation results should be interpreted.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CASA, a module that replaces the self-attention mechanism in Transformer encoders with a 1D CNN autoencoder that computes a per-token score map, applied element-wise to the value matrix. The authors claim that CASA approximates QK^T/sqrt(d_k) at linear complexity, reduces memory usage by up to 77.7%, accelerates inference by 44.0%, and achieves state-of-the-art results on eight multivariate long-term time-series forecasting benchmarks, ranking first in 54 of 64 metrics. The paper also reports experiments where CASA replaces attention in vanilla Transformer, PatchTST, and iTransformer, with improvements in 40 of 42 metrics, a robustness study across input lengths, and a correlation-matrix analysis on the Weather dataset.","tokens_in":17902,"tokens_out":5682,"duration_ms":53712,"significance":"If the theoretical framing is corrected, CASA is a potentially useful empirical contribution: a simple, model-agnostic CNN gating module with linear complexity that improves accuracy over several strong baselines, with code available and a fairly extensive evaluation across eight datasets. The formal propositions are correct but elementary; the main novelty is the module and its empirical validation, not the formal analysis. The paper would be strengthened by honest framing as a gating mechanism, by uncertainty quantification, and by consistent reporting of the abstract's claims with the full results.","major_comments":[{"comment":"The paper claims that the CNN autoencoder 'approximates QK^T/sqrt(d_k)', but the implemented operation does not have the form of an attention-score matrix product. In Eq. (8), Attention(Z_{i+1}) = softmax(Score(Z_i)) ⊛ V_{i+1}; with V_{i+1} ∈ R^{N×D} and ⊛ denoting element-wise product, Score(Z_i) is a per-token vector in R^{N×D}, not an N×N pairwise matrix. Standard self-attention in Eq. (4) produces each output row as a weighted sum of all value rows, whereas Eq. (8) only rescales each value row by a scalar gate. The convolution over variates makes the gate for row i depend on other rows, but it never aggregates or mixes values across variates. Therefore the claim that CASA approximates QK^T/sqrt(d_k), and the related narrative about repairing missing cross-variate information in attention scores, is not supported by the equations. Please reframe CASA as a CNN gating mechanism and re-derive the motivation accordingly.","section":"Section 3.4, Eq. (8)"},{"comment":"The abstract states 'ranking first in 87.5% of evaluated metrics', but Appendix B and Table 6 report that CASA achieves the best performance in 54 out of 64 metrics, which is 84.4%, not 87.5%. The abstract should be corrected to match the full results.","section":"Abstract vs. Appendix B"},{"comment":"The complexity analysis concludes that the method scales linearly in N, L, and H by treating the hidden dimension D and kernel size k as constants, but in the experimental regime D≈512, and k is a small integer, so D^2 and kD^2 terms are comparable to or larger than LD and DH for L=96 and H=720 (LD≈4.9e4, D^2≈2.6e5). The linear scaling is therefore asymptotic, not empirically demonstrated at the tested operating points. In addition, Appendix D states that the Traffic efficiency experiment uses L=512 and H=96, while the forecasting results use L=96; the memory and inference numbers in Table 1 and Figure 6 thus come from a different operating point than the accuracy experiments. Please specify the exact baseline and configuration for the claimed 77.7% memory reduction; Table 1 shows CASA at 1684 MB versus SOFTS at 1720 MB, which is only about a 2% saving.","section":"Section 3.4, Complexity Analysis and Appendix D"},{"comment":"All results are reported without standard deviations, confidence intervals, or the number of seeds. Many of the reported differences are small (for example, ETTm2 average MSE 0.276 versus 0.287 for SOFTS, and ECL average MSE 0.168 versus 0.174 for SOFTS), so a claim of state-of-the-art performance across 64 metrics needs evidence that the improvements are not due to training noise. Please add variance estimates at least for the main results table.","section":"Tables 2, 3, and 6"}],"minor_comments":[{"comment":"The word 'comtemporarlly' should be corrected to 'contemporary'.","section":"Section 4, Baselines"},{"comment":"The word 'model-agnosticically' should be corrected to 'model-agnostically'.","section":"Abstract"},{"comment":"The symbol f is used both for the affine map defining V_{i+1} and, implicitly, for the score network; please use distinct notation for these two functions.","section":"Section 3.4, Eqs. (8)-(9)"},{"comment":"Table 6 reports results for TimesNet, Stationary, and SCINet, which are not listed among the baselines in Section 4; please add the appropriate references or explain the provenance of those numbers.","section":"Table 6"},{"comment":"The text says the comparison uses 'MSE between their probability density functions (PDFs)', but the table header only says 'MSE'; please clarify the exact computation and metric name in the table.","section":"Section 4.4, Table 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth engaging, but not for the reason it gives. The CASA module is a channel-gating mechanism; Eq. (8) is an element-wise product, so it cannot reproduce the row-mixing of softmax(QK^T)V. The theoretical story in Section 3.4—that the CNN autoencoder approximates attention scores—does not survive contact with the math. What remains is a solid empirical study of a cheap, modular attention replacement, with extensive results across three tokenization schemes.\n\nThe genuinely new piece is the score network: a 1D CNN autoencoder that produces a per-token score map, combined with V element-wise. The integration with point-, patch-, and channel-wise tokenization is a useful contribution, and the experiments are broad: eight datasets, four horizons, 64 metrics, and a 54/64 best-score count. The code is public. The two propositions are correct but near-tautological—they only say affine maps act row-wise—so they add little.\n\nThe soft spots are in the framing and the reporting. First, the attention approximation claim is wrong. With Z_i in R^{N x D}, Score(Z_i) must be in R^{N x D}, and the output row i is v_i times a scalar. Convolution lets the scalar for row i depend on other rows, but no information from other variates is added to the output. That is gating, not attention. It may be a good gate, but the paper should say so. Second, the abstract says 87.5% first-place metrics; the appendix and the table say 84.4% (54/64). That is a concrete error. Third, the complexity analysis hides D^2. They call it constant and ignore it, but at D=512 and L*H ~ 69k, D^2 ~ 262k is four times larger. The linear-in-L,H claim is asymptotic, not true in the tested regime. The memory table also does not support the 77.7% headline: on Traffic, CASA is 1684 MB vs SOFTS 1720 MB—about 2%. Finally, no seeds or error bars, and baseline numbers are not tied to their sources, so the SOTA claim is not yet fully verifiable.\n\nFor whom: anyone working on efficient multivariate forecasting or attention alternatives. The module is simple enough to drop into an existing model and test. I would like to see a revision that fixes the framing and the reporting; a serious referee can sort that out. I would accept it for review, with the expectation that the authors either drop the approximation claim or justify it, correct the abstract, and report variance.","headline":"A useful empirical module whose theoretical motivation is wrong; the paper deserves a serious referee after reframing.","tokens_in":18439,"tokens_out":3750,"would_cite":true,"duration_ms":38548,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing Transformer attention scores with a CNN autoencoder makes multivariate time-series forecasting faster, lighter, and more accurate.","keywords":["multivariate long-term time series forecasting","CNN autoencoder","attention score approximation","channel-wise tokenization","cross-dimensional interactions","linear complexity","efficient Transformer"],"falsifier":"Run the Traffic benchmark at $L=96$, $H=720$, batch size 16 and measure CASA's memory and latency against iTransformer and SOFTS on identical hardware; if the reduction is only the roughly 2% gap shown in Table 1 rather than the claimed 77.7%, the central efficiency claim fails. Separately, if CASA's advantage vanishes on a dataset with weak inter-variate correlations, the cross-dimensional mechanism is not the cause.","tokens_in":17404,"feed_emoji":"📈","tokens_out":8292,"duration_ms":71333,"temperature":0.7,"pith_summary":"This paper aims to establish that the attention score matrix $QK^\\top/\\sqrt{d_k}$ in Transformers can be replaced by a CNN autoencoder that learns the score map directly, and that this substitution improves both efficiency and accuracy for multivariate long-term time-series forecasting. The authors argue that standard query and key embeddings are variate-independent under channel-wise tokenization, and time-independent under point/patch tokenization, so the attention score never uses correlations between variates; the CNN score network restores that information by treating variates as channels. On eight real-world datasets they report 54 of 64 best MSE/MAE scores, up to 77.7% memory reduction, and 44.0% faster inference, with the module working across point-wise, patch-wise, and channel-wise Transformers. The practical stakes are that attention is the main computational bottleneck in forecasting Transformers, and a drop-in linear-complexity replacement that also improves accuracy would remove a major cost barrier.","feed_headline":"CNN autoencoder replaces attention scores for faster forecasting","feed_subtitle":"A drop-in module ranks first in 54 of 64 multivariate forecasting benchmarks while cutting memory use.","key_machinery":"The central object is the score network: a 1D CNN autoencoder with an inverted bottleneck, used in place of $QK^\\top/\\sqrt{d_k}$. It takes the embedded feature $Z_i \\in \\mathbb{R}^{N\\times D}$ with variates as channels, expands it through convolution into a high-dimensional latent space, then compresses back to an attention score map; softmax of that map is multiplied element-wise by the value embedding. This carries the argument because it makes query/key construction variate-dependent and keeps the overall complexity $O(NL + NLD + ND^2 + NkD^2 + NDH)$, which scales linearly in $N$, $L$, and $H$ when the hidden dimension $D$ and kernel size $k$ are treated as constants.","core_discovery":"CASA treats each variate as a channel and replaces the affine query/key embeddings of self-attention with a 1D CNN autoencoder that approximates $QK^\\top/\\sqrt{d_k}$; the attention output is then softmax of the learned score, multiplied element-wise by the value embedding. The paper proves that with channel-wise tokenization the standard query and key embeddings are variate-independent (Proposition 1), and with point-wise or patch-wise tokenization they are time-independent (Proposition 2), so conventional attention scores are built without cross-variate information. The CNN score network, built as an inverted-bottleneck autoencoder, expands all variates into a high-dimensional latent space and then compresses channels, making score construction variate-dependent at claimed linear cost in $N$, $L$, and $H$. In the authors' experiments, swapping self-attention for CASA improved 40 of 42 metrics across vanilla Transformer, PatchTST, and iTransformer, and the full CASA model ranked first in 54 of 64 metrics across eight datasets.","pith_inferences":["The paper only evaluates forecasting; the same score-network substitution could plausibly extend to other Transformer uses where the score matrix is smooth and the token count is large, such as long-document processing or multivariate anomaly detection.","The linear-complexity claim is asymptotic: because $D$ and $k$ are treated as constants, the practical linear scaling only appears once $L$ and $H$ are much larger than $D^2$, which is not guaranteed in the tested settings.","A direct mechanistic test would be to train CASA on synthetic series with known but weak inter-variate correlations; if the accuracy gain over standard attention disappears, the cross-dimensional score mechanism is the active ingredient."],"forward_implications":["Transformer encoders for multivariate series can be made linear in variate count, input length, and horizon without sacrificing accuracy; the paper reports first place in 54 of 64 metrics across eight datasets.","The module is a drop-in replacement: swapping self-attention for CASA improved 40 of 42 evaluated metrics across vanilla Transformer, PatchTST, and iTransformer, so no change in tokenization is required.","On high-variate data such as Traffic with 862 channels, CASA's predicted inter-variate correlation matrix matches the ground truth more closely than the compared baselines, indicating the score network preserves cross-variate structure.","Memory and latency shrink relative to token-based attention at fixed hardware, which makes longer lookback windows and longer forecast horizons more affordable in practice."],"supporting_citations":[{"why":"Defines the self-attention score $QK^\\top/\\sqrt{d_k}$ that CASA approximates.","marker":"[Vaswani, 2017]"},{"why":"Supplies the channel-wise tokenization backbone (iTransformer) that CASA replaces attention in and treats as a baseline.","marker":"[Liu et al., 2023]"},{"why":"SOFTS is the previous state-of-the-art baseline whose preprocessing and results CASA builds on and compares against.","marker":"[Han et al., 2024]"},{"why":"PatchTST provides the patch-wise tokenization baseline and one integration target for swapping in CASA.","marker":"[Nie et al., 2022]"},{"why":"Supports the claim that channel-wise tokenization performs best among tokenization schemes, motivating CASA's design.","marker":"[Yu et al., 2024]"},{"why":"Inspires the inverted-bottleneck autoencoder structure used for the score network.","marker":"[Wilson et al., 2016]"},{"why":"Supports the idea that embedding low-dimensional features into a high-dimensional latent space improves expressiveness.","marker":"[Bengio et al., 2013]"},{"why":"Supplies the eight long-term forecasting datasets used in all experiments.","marker":"[Zhou et al., 2021]"}],"fun_headline_variants":["CNN autoencoder cuts attention compute and tops forecasting benchmarks","CNN autoencoder score attention cuts compute 77.7%, speeds up 44%","Drop-in CNN autoencoder makes attention variate-aware, top in 54/64 metrics","Linear-cost attention scores via CNN autoencoder, SOTA on 8 datasets","Variate-dependent score learning: CNN autoencoder reduces compute 77.7%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The efficiency story assumes the hidden dimension and kernel width are small fixed constants, but in the reported experiments those constants are large enough that quadratic terms can dominate, and the headline 77.7% memory saving is not tied to a clear baseline in the provided tables.","fun_headline_variants_meta":{"raw":{"variants":["CNN autoencoder cuts attention compute and tops forecasting benchmarks","CNN autoencoder score attention cuts compute 77.7%, speeds up 44%","Drop-in CNN autoencoder makes attention variate-aware, top in 54/64 metrics","Linear-cost attention scores via CNN autoencoder, SOTA on 8 datasets","Variate-dependent score learning: CNN autoencoder reduces compute 77.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001709,"raw_usage":{"total_tokens":6745,"prompt_tokens":907,"completion_tokens":5838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":5736}},"tokens_in":523,"tokens_out":5838,"duration_ms":42634,"temperature":1.0,"reasoning_tokens":5736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:03:37.738737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Traffic benchmark at $L=96$, $H=720$, batch size 16 and measure CASA's memory and latency against iTransformer and SOFTS on identical hardware; if the reduction is only the roughly 2% gap shown in Table 1 rather than the claimed 77.7%, the central efficiency claim fails. Separately, if CASA's advantage vanishes on a dataset with weak inter-variate correlations, the cross-dimensional mechanism is not the cause.","supporting_citations":[{"cited_title":"itransformer: Inverted transformers are effective for time series forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies the channel-wise tokenization backbone (iTransformer) that CASA replaces attention in and treats as a baseline."},{"cited_title":"Softs: Efficient multivariate time series forecasting with series-core fusion","cited_arxiv_id":null,"evidence_quote":"SOFTS is the previous state-of-the-art baseline whose preprocessing and results CASA builds on and compares against."},{"cited_title":"A time series is worth 64 words: Long-term forecasting with transformers","cited_arxiv_id":null,"evidence_quote":"PatchTST provides the patch-wise tokenization baseline and one integration target for swapping in CASA."},{"cited_title":"Deep kernel learn- ing","cited_arxiv_id":null,"evidence_quote":"Inspires the inverted-bottleneck autoencoder structure used for the score network."},{"cited_title":"Representation learning: A review and new perspectives","cited_arxiv_id":null,"evidence_quote":"Supports the idea that embedding low-dimensional features into a high-dimensional latent space improves expressiveness."}],"review_version":1}