{"id":"03598e9c-9fb7-4537-a206-a4020e29e0d1","arxiv_id":"2411.13264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A sparse-attention transformer that picks the top past time steps and masks variables one at a time outperforms a fixed-lag VAR baseline for Granger causal discovery on synthetic linear time series.","lead":"This paper proposes a transformer that selects the most informative past time steps and then masks each variable in turn to infer Granger causal links in multivariate time series. On synthetic data with small random delays, it reports higher causal discovery accuracy than a fixed-lag vector autoregression baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The temporal top-k selection is biased by the causal mask: column j has only 2k−j+1 non-masked entries, so the 'scaling factor of 2k' argument in Sec. 4 is false and older lags are systematically favored.","rationale":"The paper's central claim is that the model can contextually select the most important past time instances, in contrast to a fixed-lag VAR. That claim rests entirely on the column-sum criterion in Sec. 4. My concern is not merely that attention weights may fail to encode causal importance—an interpretive caveat the reader already flagged—but that the column sums are not even commensurable under the causal mask. Because the number of non-masked entries per column ranges from 1 to 2k, the raw column sums have a systematic bias toward older time instances. This is an internal inconsistency in the derivation, not a disagreement with an outside consensus. The proposed concrete test would settle whether this bias drives the results. If it does, the reported AUC-ROC/F1 values cannot be interpreted as evidence for the claimed sparse-attention lag selection; the method reduces to a heuristic fixed window with an arbitrary top-k rule. I also note that even after correcting the normalization, the hard top-k selection is non-differentiable, so the temporal attention parameters may receive no gradient; that is a further reason the 'contextual' learning claim is unsupported. The other weaknesses noted by the reader (no error bars, limited baselines, no code) are secondary; this one attacks the mechanism itself. Therefore I would adjust the verdict from CONDITIONAL to REJECT for the current manuscript, while acknowledging that a revision with normalized column means, differentiable selection, and re-evaluation could change that assessment.","tokens_in":7894,"tokens_out":8166,"duration_ms":89247,"concrete_test":"Analytical check: with k=10 and uniform attention (A_ij=1/i for j≤i), compute s_j for j=1..20; the top-10 set is the 10 oldest indices for every possible dataset, independent of the true lags. Empirical check: re-run the full pipeline replacing raw column sums s_j with normalized column means s_j/(2k−j+1) before choosing top-k; compare the selected time-index sets and AUC-ROC/F1 on the same 20 synthetic datasets. If normalized selection changes the selected lags or the reported scores materially, the published results are driven by mask-induced count bias rather than by learned temporal importance.","verdict_should_be":"REJECT","load_bearing_attack":"In Sec. 4, the temporal attention matrix A_t is computed as softmax(Q1K1^T/√d + M), where M has -∞ above the diagonal and 0 on and below it. Thus column j contains nonzero entries only in rows i=j,...,2k, i.e., 2k−j+1 entries. Because every row of A_t sums to 1, raw column sums are not comparable: older columns (small j) receive contributions from more rows. The paper states that comparing column sums is equivalent to comparing column means because they differ only by the constant 2k, but the number of non-masked entries per column is not constant. Even for uniform attention with A_ij=1/i for j≤i, s_j=H_{2k}−H_{j−1}, so s_j strictly decreases with j and the top-k columns are always the oldest k time instances. The claimed ability to 'pick the most significant past time instances' is therefore not established by the reported procedure; the selection is confounded by mask-induced count bias, and the Granger indices computed from the selected lags inherit that bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Sparse Attention Transformer (SAT) for Granger causal discovery in multivariate time series with small, variable lags. The model first applies temporal self-attention over a 2k-length window with a causal mask, selects the k time instances with the largest column sums, and then applies inter-variable attention with leave-one-variable-out masking to produce Granger causality indices. The method is evaluated on synthetic linear datasets with up to 10 variables and 150 time steps, using AUC-ROC and F1 scores against a VAR-based Granger causality baseline. The paper claims that SAT significantly outperforms VAR and that the top-k temporal selection removes the need to specify a fixed lag manually.","tokens_in":8188,"tokens_out":4197,"duration_ms":53068,"significance":"If the central claims held, the paper would offer a practical attention-based alternative to fixed-lag VAR for Granger causal discovery, with a concrete algorithm and a falsifiable evaluation protocol on synthetic benchmarks. The two-stage attention design is clearly specified, and the authors provide a pseudocode algorithm and report results across four dataset groups. However, the main novelty — context-dependent selection of the most important past time instances — is undermined by a mask-induced bias in the column-sum comparison, and the empirical comparison with VAR is not statistically supported as written. The significance therefore depends on whether the temporal-selection mechanism can be corrected and the evaluation strengthened.","major_comments":[{"comment":"The claim that comparing column sums is equivalent to comparing column means because they differ only by a scaling factor of 2k is false under the causal mask. In the temporal attention matrix A_t, entry (i,j) is nonzero only for i >= j, so column j contains 2k - j + 1 active entries, not 2k. For uniform attention with A_ij = 1/i, the column sum is H_{2k} - H_{j-1}, which strictly decreases with j; the top-k selection would then always choose the oldest k time steps. This count bias confounds the temporal selection and invalidates the paper's central claim that the model contextually picks the most significant past time instances. The authors should use a mask-aware normalization, such as averaging over the unmasked entries, or otherwise demonstrate that the selection is not driven by the mask.","section":"§4, temporal attention and Algorithm 1"},{"comment":"The statement that SAT 'performs significantly better than the baseline method' is not supported statistically. No standard deviations, confidence intervals, or hypothesis tests are reported across the 20 datasets per group. In addition, the VAR baseline uses 20 lags on series of length 150 with up to 10 variables, which is severely overparameterized (roughly 20D coefficients per equation) and will underperform for reasons unrelated to Granger causal structure. A fair comparison requires a lag-selection procedure for VAR, such as BIC/AIC, or a lag length appropriate to the sample size, together with repeated-sampling variability measures.","section":"§6 and Table 1"},{"comment":"The unrestricted-model errors appear to be computed on the training data: the pseudocode stores Y' from the backpropagation phase and the restricted-model predictions are also obtained on the same data. Training residuals can be made very small by overfitting, which makes the Granger index ln(sigma_R^2 / sigma_U^2) uninformative and potentially inflated. Evaluation of both unrestricted and restricted models should be performed on held-out data, and the authors should report the actual residual variances.","section":"§4, Algorithm 1"},{"comment":"The setup sets k=10 because the synthetic data has a true maximum lag of 10 and the window size is 2k=20. This means the claimed advantage over 'manually feeding a fixed time lag value' is partly assumed rather than demonstrated. The model still requires the maximum lag / window size as a hyperparameter, and the experiments do not test robustness to misspecified k, lags larger than k, or variable lags with a wider range. At minimum, the paper should distinguish the known maximum lag used to construct the window from the unknown per-instance lags the model is supposed to select.","section":"§5.1"},{"comment":"Masking a column of A_d with -infinity before softmax forces that variable's attention weight to zero in every row, but the paper does not establish that this operation is equivalent to Granger's restricted regression model. Attention weights are not shown to correspond to regression coefficients, and removing one variable also changes the softmax normalization for all other variables. As a concrete correctness check, I suggest a controlled experiment: when a variable that has no true causal effect is masked, the average CGCI for that variable should be near zero; if this does not hold, the proposed index measures model sensitivity rather than Granger causality.","section":"§4, inter-variable attention and Granger masking"}],"minor_comments":[{"comment":"The caption says 'across the three groups of data samples' but the table reports four groups (D=4, 5, 6, 10); please correct the caption.","section":"Table 1 caption"},{"comment":"There is a typo: 'Autocorrelaion' should be 'Autocorrelation'.","section":"§3.1"},{"comment":"In the computation of s_l, the indices are inconsistent: s_l is written as a sum over j of A_{j,l}, but the loop variable and the later reference to 's_k values' should be aligned. Please rewrite the loop with clear row/column indices.","section":"Algorithm 1"},{"comment":"The positional embedding is mentioned only in the training details; the architecture description in §4 should state where positional embeddings are injected, since the temporal attention operates on the raw window.","section":"§5.2"},{"comment":"The phrase 'significantly better' should be replaced by a quantitative statement backed by confidence intervals or statistical tests, as noted in the major comments.","section":"§6"},{"comment":"The contribution statement says the model 'picks the top-k time-instances contextually,' but the subsequent description shows that k is a fixed hyperparameter equal to the true maximum lag in the experiments; the text should acknowledge this limitation.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a compact conference submission with a clear algorithm and synthetic evaluation, but the central temporal-selection mechanism contains a mask-count bias that appears to invalidate the claimed novelty as written. This is a load-bearing issue rather than a presentation concern. With mask-aware normalization, held-out evaluation, and a fair VAR baseline, the approach may become defensible, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — read this one mainly as a cautionary example. The exact mechanism, top-k temporal lag selection via attention column sums followed by variable-masking attention for Granger indices, is new relative to the cited literature, and the paper is clearly written and easy to reproduce from the algorithm block. The synthetic C4C-style setup and the comparison against a fixed-lag VAR are appropriate for a first test. The citation pattern is fine; the related work section is short but relevant.\n\nThe load-bearing problem is in Section 4. The paper says column sums of the temporal attention matrix can be compared because they differ only by a scaling factor of 2k. That is false. The causal mask leaves only 2k-j+1 non-masked entries in column j. Even with uniform attention, column sums are H_{2k}-H_{j-1}, strictly decreasing in j, so the top-k selection always picks the oldest k lags. The stress-test note is correct. The claimed ability to \"pick the most significant past time instances\" is not demonstrated by the reported procedure, and the Granger indices computed after selection inherit the bias.\n\nSecondary problems are real. The unrestricted-model errors appear to be computed on the training data, so the Granger indices may reflect overfitting rather than generalization. No standard deviations or significance tests are reported, and the VAR baseline with 20 lags on 150 steps is overparameterized. The experiment sets k=10 exactly equal to the known maximum lag, so it never tests the claimed advantage when the lag is unknown. No code is provided.\n\nThe idea is not without merit. A corrected version that normalizes column sums by the number of valid entries, evaluates on held-out data, and includes error bars would be worth another look. As written, the central novelty rests on a false normalization, and the evaluation is too weak to rescue it.\n\nWho this is for: researchers interested in attention-based causal discovery, mainly as a demonstration of how easily attention-mask geometry can confound interpretability claims. I would not cite it in its current form. For peer review: I would have a serious referee look at it rather than desk-reject, because the flaw is precise and fixable, but I would not accept this version.","headline":"A clean, short sparse-attention Granger paper whose key lag-selection step is invalidated by the causal mask, and whose evaluation would not support the claims even without that flaw.","tokens_in":8652,"tokens_out":3956,"would_cite":false,"duration_ms":41493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a transformer with sparse attention can recover Granger causal links in multivariate time series without a fixed lag, by learning which past time instances matter most.","keywords":["Multi-variate time series","Granger Causality","Stationarity","Self-attention","Sparse Attention","Transformers","Causal discovery","Variable time lags"],"falsifier":"Generate a linear bivariate process where $X_t$ depends only on $Y_{t-10}$ plus small noise, train the Sparse Attention Transformer with window $2k = 20$ over many random seeds, and check the selected top-10 index set; if the true lag 10 is not consistently selected yet a VAR fit with the true lag detects the edge, the column-sum importance criterion has failed.","tokens_in":7721,"feed_emoji":"🔗","tokens_out":7823,"duration_ms":75165,"temperature":0.7,"pith_summary":"This paper tries to establish that a transformer whose attention mechanism is made sparse can recover Granger causality—whether past values of one series improve prediction of another—among variables in multivariate time series when the delays between cause and effect are small, unknown, and vary over time. The model first runs temporal attention over a sliding window, ranks past time instances by the column sums of the attention matrix, and keeps only the top k as the effective lag set; it then runs attention across variables and compares prediction errors with and without each variable masked. If the method works, causal discovery no longer needs a fixed lag specified in advance, which matters for real series where delays drift, such as climate or crop-yield data with shifting monsoon onsets. On synthetic time-homogeneous linear datasets with random delays, the paper reports that its Sparse Attention Transformer beats a fixed-lag vector autoregression (VAR) Granger baseline on AUC-ROC for all tested variable counts and on F1 for most.","feed_headline":"Sparse-attention transformer beats fixed-lag Granger causality","feed_subtitle":"It learns which past time steps matter, reporting higher AUC-ROC and F1 on synthetic time series with random delays.","key_machinery":"The carrying mechanism is the column-sum selection rule applied to the temporal attention matrix. In the temporal stage, self-attention is computed over a window of 2k past time steps with a causal mask; because each row of the resulting lower-triangular matrix sums to one, the sum of a column measures the average influence that one past time step exerts on all later tokens, and comparing column sums is equivalent to comparing column means. The paper takes the k columns with the largest sums as the model's learned choice of effective lags, and this selection is what allows the method to avoid a fixed lag. The second stage multiplies the selected steps' attention across variables and derives Granger indices by masking one variable column at a time during inference.","core_discovery":"The paper's central claim is that unknown, variable lags can be handled by a two-stage sparse attention mechanism rather than by fixing a lag in advance. Stage one treats the recent past as a window of 2k time tokens, applies a causal mask, and computes a temporal attention matrix; the column sums of this matrix are taken as importance scores, and the k columns with the largest sums select which past time instances form the effective input. Stage two transposes those k selected steps so the D variables become tokens, trains to predict the current vector, and at inference masks one variable at a time to obtain restricted predictions. The Granger indices are log ratios of restricted to unrestricted prediction error variances, normalized into a causation matrix. On synthetic linear, time-homogeneous datasets with maximum lag 10 (window 20, top k = 10), the paper reports that this model outperforms VAR-based Granger causality with the same window on AUC-ROC for D = 4, 5, 6, 10 and on F1 except D = 6.","pith_inferences":["If the column-sum interpretation is correct, the same top-k temporal selection could be used as a lag-selection preprocessor for standard Granger tests, giving classical methods an adaptive lag instead of a fixed one.","The paper leaves non-linear and non-stationary series to future work; a natural stress test is whether the column-sum ranking remains stable when the underlying causal strength changes over time, since attention statistics then shift.","The reported comparison uses the same window length for the VAR baseline; an oracle-lag VAR or a VAR with information-criterion lag selection would clarify how much of the gap comes from adaptive lag choice rather than from attention itself.","If the model genuinely learns context-dependent lags, the selected top-k index set should vary across windows when delays vary; measuring that variability would directly test whether the method is adapting rather than memorizing fixed positions."],"forward_implications":["Practitioners can run Granger causal discovery on stationary multivariate series without choosing a lag in advance; the top-k selection supplies the effective past times contextually.","On series with random or shifting delays, the method reports higher AUC-ROC and F1 than fixed-lag VAR Granger causality on the synthetic benchmarks tested.","Because only k of the 2k window positions are retained and each variable is masked one at a time, the attention computation is sparse and does not require full pairwise attention across all time steps.","The output is a normalized D by D causation matrix comparable with ground-truth causal graphs and with classical Granger outputs, so existing evaluation metrics transfer directly."],"supporting_citations":[{"why":"Supplies the restricted VAR model and the conditional Granger causality index used as the paper's baseline and evaluation.","marker":"[12]"},{"why":"Defines scaled dot-product self-attention, the base operation for both attention stages.","marker":"[14]"},{"why":"Provides the premise that self-attention weights can be interpreted as indicators of causal links.","marker":"[8]"},{"why":"Supplies the synthetic data generation framework used to build the benchmark datasets with random delays.","marker":"[10]"},{"why":"The prior spatial-temporal causal transformer on fixed-lag data that this method extends by adding sparse selection.","marker":"[22]"},{"why":"The ProbSparse attention architecture that the paper contrasts with its own top-k column-sum selection.","marker":"[21]"},{"why":"A causal transformer built on ProbSparse attention whose mechanism the paper distinguishes from its own.","marker":"[17]"},{"why":"Shows attention weights can yield instance-wise Granger causality in event sequences, supporting the attention-as-causality premise.","marker":"[16]"}],"fun_headline_variants":["Sparse attention picks lags, beats fixed-lag VAR","Adaptive lag selection via sparse attention outperforms VAR","Transformer sparse attention learns Granger lags automatically","Selective sparse attention improves Granger causality inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the unproven assumption that the column sums of the temporal attention matrix rank which past time instances matter for predicting the output; if that ranking is wrong, the selected lags and the resulting Granger indices lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Sparse attention picks lags, beats fixed-lag VAR","Adaptive lag selection via sparse attention outperforms VAR","Transformer sparse attention learns Granger lags automatically","Selective sparse attention improves Granger causality inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2661,"prompt_tokens":930,"completion_tokens":1731,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1668}},"tokens_in":546,"tokens_out":1731,"duration_ms":12069,"temperature":1.0,"reasoning_tokens":1668,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:38:10.853252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a linear bivariate process where $X_t$ depends only on $Y_{t-10}$ plus small noise, train the Sparse Attention Transformer with window $2k = 20$ over many random seeds, and check the selected top-10 index set; if the true lag 10 is not consistently selected yet a VAR fit with the true lag detects the edge, the column-sum importance criterion has failed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the restricted VAR model and the conditional Granger causality index used as the paper's baseline and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the premise that self-attention weights can be interpreted as indicators of causal links."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the synthetic data generation framework used to build the benchmark datasets with random delays."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior spatial-temporal causal transformer on fixed-lag data that this method extends by adding sparse selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A causal transformer built on ProbSparse attention whose mechanism the paper distinguishes from its own."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows attention weights can yield instance-wise Granger causality in event sequences, supporting the attention-as-causality premise."}],"review_version":1}