Pith. sign in

REVIEW 3 major objections 6 minor 24 references

XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Replacing dot-product attention with a rank-correlation score improves time-series forecasting up to 9.1 percent.

desk verdict A genuinely new attention scoring function built on a differentiable Chatterjee xi, but the empirical support is mixed and the 'sample' interpretation is shaky. read the letter →

arxiv 2506.02694 v1 pith:WLR2CUEC submitted 2025-06-03 cs.LG cs.AI

classification cs.LGcs.AI
keywords timeseriesforecastingTransformerattentionmechanismChatterjeecorrelationcoefficientranknonlineardependencedifferentiablesortingSoftSort
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes XicorAttention, a new attention mechanism for Transformer-based multivariate time series forecasting. Standard attention scores are computed by a dot product between query and key vectors, which detects only linear relationships; the paper argues that time series contain nonlinear dependencies that this linear score misses. The authors replace the dot-product score with a differentiable approximation of Chatterjee's rank correlation coefficient $\xi$, which measures how well one variable predicts another through arbitrary monotone or nonlinear functions. Integrated into PatchTST, iTransformer, and TimeXer, the mechanism improves forecasting accuracy on real-world benchmarks by up to approximately 9.1 percent compared to the original models.

What carries the argument

The central object is Chatterjee's rank correlation coefficient $\xi_n(X,Y)$, defined as $1 - \frac{3\sum_{i=1}^{n-1}|r_{i+1}-r_i|}{n^2-1}$, where the pairs are sorted by $X$ and $r_i$ are the ranks of the corresponding $Y$ values. It converges to 0 if $X$ and $Y$ are independent and to 1 if $Y$ is a measurable function of $X$. To make it usable inside a Transformer, the paper approximates the sorting permutation with SoftSort and the ranking operation with FastSoftRank's projection onto the permutahedron, so gradients can flow back into the query and key projection matrices.

What would settle it

Train a Transformer model with XicorAttention on the same benchmarks while randomly permuting the coordinate positions of query and key vectors identically within each attention head before computing the $\xi$ score; if forecasting accuracy is unchanged, the mechanism merely rewards coordinate alignment and not genuine nonlinear dependence.

Watch

Extended reading notes

Core claim

The central claim is that computing attention weights with Chatterjee's $\xi$ correlation coefficient, rather than with the scaled dot product, lets Transformer models capture nonlinear query-key dependencies and thereby improves multivariate long-term time-series forecasting. Concretely, the paper replaces $QK^\top/\sqrt{d}$ in each attention head with $\xi_d(q,k)$, treating the $d$ coordinates of each query-key pair as paired samples. Because Chatterjee's $\xi$ is defined through sorting and ranking operations and is not differentiable, the paper builds a differentiable surrogate using SoftSort for the permutation step and FastSoftRank for the ranking step, with a straight-through trick to keep exact ordering in the forward pass. Experiments across ETT, Exchange, Weather, Electricity, and Traffic show that replacing the attention layer of PatchTST yields consistent gains, with the largest improvement being 9.12 percent on the Exchange dataset; gains are smaller or negative when the method is applied to models that emphasize inter-variable relationships, which the authors attribute to $\xi$'s weaker power on smooth, nearly linear dependencies.

Load-bearing premise

The paper assumes the coordinates of each query and key vector within a head behave like paired random samples from a joint distribution, so that a correlation statistic computed across those coordinates measures the true dependence between query and key; if the coordinates instead encode positional or structural information, the whole computation measures an artifact of coordinate ordering.

Editorial extensions

If this is right

  • Any Transformer model whose attention layer is replaced by XicorAttention inherits a score that detects nonlinear and oscillatory dependencies, not just linear ones, at $O(n^2)$ complexity.
  • For temporal-patching models like PatchTST, the mechanism gives consistent accuracy improvements across most tested datasets and horizons, so it can serve as a drop-in attention module for that model family.
  • Head dimension $d$ acts as the sample size for the correlation estimate, and the paper's sensitivity analysis says accuracy stabilizes for $d \ge 128$, giving a concrete design rule for multi-head configurations.
  • Avoiding the method for inter-variable attention on datasets with strong linear correlations, such as Traffic with 862 variates, prevents the observed performance degradation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's diagnosis suggests a broader testable claim: attention mechanisms that measure only linear association underperform specifically on forecasting tasks where the relevant query-key dependence is nonlinear, and a correlation-based score should be most helpful when patched inputs or variates have oscillatory or threshold-like relationships.
  • The interpretation of $\xi_d(q,k)$ as a dependence estimate rests on treating the $d$ coordinates of each attention head as exchangeable paired samples; for models where coordinates encode positional structure or patch order, the same computation could reward artifacts of coordinate arrangement rather than semantic relevance. An ablation that permutes coordinates within each head before computing
  • Since the gain concentrates in PatchTST and degrades in iTransformer on Traffic, a natural next experiment is an adaptive scheme that selects between dot-product and $\xi$ per head or per dataset based on an estimated linearity of the dependency, rather than replacing attention uniformly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes XicorAttention, a replacement for the scaled dot-product in Transformer attention that uses a differentiable approximation of Chatterjee's rank correlation coefficient ξ. The authors employ SoftSort and FastSoftRank to make the sorting and ranking operations differentiable and integrate the mechanism into PatchTST, iTransformer, and TimeXer. The experiments cover seven multivariate time-series datasets with four forecasting horizons, and the abstract claims an improvement of up to approximately 9.1% over existing models. The paper reports that the integrated models achieve the best or second-best results on most datasets, but also documents substantial degradations, notably a 37.5% MSE increase when applied to iTransformer on Traffic.

Significance. The idea of using a rank-based nonlinear dependence measure inside attention is novel and could be relevant to time-series forecasting. The differentiable relaxation of sorting and ranking is technically sound and clearly presented. However, the statistical interpretation of computing ξ over the head-dimension coordinates as an estimate of dependence between query and key is not justified, and the empirical evidence for consistent improvement is mixed. If the statistical foundation can be repaired and the claims appropriately qualified, the work could be a useful empirical contribution. As it stands, the central claim at the level stated in the abstract is not supported.

major comments (3)
  1. [§3.1–3.2, Eq. (5)] The paper treats the d coordinate pairs (q_i, k_i) of the query and key vectors as i.i.d. samples for computing Chatterjee's ξ_n, stating in §3.2 that n and d are used interchangeably. This identification is invalid: q and k are deterministic learned projections of the same input, and the coordinates are not samples from any joint distribution. Consequently, ξ_d(q,k) is not a consistent estimator of a population dependence, and the claimed property of capturing nonlinear dependencies inherent in time series data does not follow from Chatterjee's theory. The authors should either provide a rigorous statistical model under which the coordinates are exchangeable samples, or reframe the method as a heuristic nonlinear similarity measure and adjust the motivation accordingly.
  2. [§5.2, Table 5; abstract] The abstract's claim that incorporating nonlinear correlation into the attention improves forecasting accuracy is not supported by the full set of results. On Traffic, replacing iTransformer's attention degrades MSE by 37.5% (from 0.422 to 0.570), and multiple other cells show negative enhancement, such as ETTh1 TimeXer MSE (-5.15%) and Electricity TimeXer MSE (-5.12%). The paper's acknowledgement of this behavior in Sections 5.2 and 6.1 should also be reflected in the abstract and in the 'consistently outperforms' statement in Section 5.1.
  3. [§6.2, Figure 3] The recommendation of a head dimension d≥128 is derived post hoc from the same sensitivity experiments used to motivate it. Since the paper identifies d with the sample size n for ξ, this recommendation conflates a heuristic architecture choice with a statistical claim about estimation accuracy. The authors should explicitly state that this is an empirical heuristic without a statistical justification.
minor comments (6)
  1. [§3.3] The phrase 'To archive this sorting operation' should be 'To achieve this sorting operation.'
  2. [§3.4] In the sentence explaining the permutahedron, 'the the elements' should be 'the elements.'
  3. [Figure 2 caption and §6.1] The Figure 2 caption says the heatmap shows the Traffic dataset, while the text in Section 6.1 says it shows the Weather dataset; please make these consistent.
  4. [Table 4] The header row of Table 4 is corrupted (e.g., 'Xicor+TimeXTimeXer', 'iT ransiTransformer'); please fix the formatting and column labels.
  5. [§3.3, Eq. (10)] The specific distance function d(x,y) used in SoftSort (e.g., L1 or L2) is not specified; please report the exact choice used in the experiments.
  6. [Throughout] The paper inconsistently abbreviates iTransformer as 'iTrans' in several places; please standardize the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the forecasting improvements are measured empirically on held-out test sets, and the statistical properties of Chatterjee's xi are imported from external, non-self-cited work.

full rationale

The paper's central claim is that replacing scaled dot-product attention with a differentiable approximation of Chatterjee's xi improves multivariate time-series forecasting accuracy. This is an empirical architectural modification, not a derivation in which the output is equivalent to the input. The statistical properties of xi (e.g., xi = 0 under independence, xi = 1 for a measurable function) are cited from Chatterjee (2021) and Dette et al. (2013), which are external works with no author overlap. The differentiable sorting and ranking machinery is taken from Prillo & Eisenschlos (2020) and Blondel et al. (2020), also external. The claimed 9.1% improvement is a reported maximum across held-out benchmark evaluations, not a quantity constructed by definition from a fitted parameter. The identification of attention-head dimension d with the sample size n in Sections 3.1 and 3.2 is a modeling assumption that may be statistically questionable, but it is not circular: xi_n is not defined in terms of forecasting error, and forecasting error is not defined in terms of xi_n. The recommendation d >= 128 in Section 6.2 is a post-hoc observation from sensitivity experiments rather than a load-bearing prediction derived from the method's equations. There are no load-bearing self-citations, and no fitted input is renamed as a prediction. Accordingly, no circularity step can be exhibited by the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its free parameters are the temperature and regularization hyperparameters of the differentiable sorting and ranking approximations, plus the head dimension which is tuned. The main burden is the unstated i.i.d.-coordinates assumption and the assumption that finite-d xi behaves like its asymptotic limit.

free parameters (3)
  • SoftSort temperature tau = not specified
    Controls the sharpness of the soft permutation matrix in Eq. (10); its value is never reported, yet it directly affects the gradient of the attention scores.
  • FastSoftRank regularization epsilon = not specified
    Controls the smoothness of the rank approximation in Eq. (14); never reported in the paper.
  • Attention head dimension d (sample size n for xi) = recommended >= 128
    Section 6.2 shows accuracy depends strongly on d; the recommendation d>=128 is fitted post hoc from the sensitivity experiments on ETTh1 and Electricity.
assumptions (4)
  • domain assumption The d coordinates of a query/key vector pair (q_i, k_i) are treated as an i.i.d. sample from a joint distribution.
    Section 3.1-3.2 use n and d interchangeably, so xi_n(q,k) is computed over coordinate pairs; no justification is given for exchangeability of embedding dimensions.
  • domain assumption The asymptotic limiting properties of Chatterjee's xi (0 iff independence, 1 iff deterministic relation) are assumed to hold approximately at finite head dimensions.
    Section 6.2 relies on 'larger sample sizes improve estimation' to recommend d>=128, implying finite-d behavior follows the asymptotic theory.
  • standard math SoftSort and FastSoftRank provide faithful differentiable surrogates for sorting and ranking, with straight-through gradients that permit stable training.
    These are existing methods cited from [15] and [2]; the paper does not re-derive them.
  • domain assumption Using xi as an attention score, followed by the model's existing normalization, still produces valid attention weights that can aggregate values.
    The paper never writes the full attention equation after xi is computed, so it implicitly assumes the rest of the attention pipeline, including softmax over keys if used, remains valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation." pith.science (2026). https://pith.science/paper/WLR2CUEC

@misc{pith2026250602694,
  author       = {Pith},
  title        = {Pith review of: XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WLR2CUEC}},
  note         = {Machine review of arXiv:2506.02694}
}
read the original abstract

Various Transformer-based models have been proposed for time series forecasting. These models leverage the self-attention mechanism to capture long-term temporal or variate dependencies in sequences. Existing methods can be divided into two approaches: (1) reducing computational cost of attention by making the calculations sparse, and (2) reshaping the input data to aggregate temporal features. However, existing attention mechanisms may not adequately capture inherent nonlinear dependencies present in time series data, leaving room for improvement. In this study, we propose a novel attention mechanism based on Chatterjee's rank correlation coefficient, which measures nonlinear dependencies between variables. Specifically, we replace the matrix multiplication in standard attention mechanisms with this rank coefficient to measure the query-key relationship. Since computing Chatterjee's correlation coefficient involves sorting and ranking operations, we introduce a differentiable approximation employing SoftSort and SoftRank. Our proposed mechanism, ``XicorAttention,'' integrates it into several state-of-the-art Transformer models. Experimental results on real-world datasets demonstrate that incorporating nonlinear correlation into the attention improves forecasting accuracy by up to approximately 9.1\% compared to existing models.

Figures

Figures reproduced from arXiv: 2506.02694 by the authors.

Figure 1
Figure 1. Overview of the proposed XicorAttention architecture integrated into an encoder-based Transformer forecasting model (left). In XicorAttention (right), atten￾tion weights are computed based on Chatterjee’s correlation coefficient ξn(q, k) be￾tween queries and keys. Since ξn involves non-differentiable sorting and ranking opera￾tions, we introduce differentiable approximations using SoftSort and SoftRank to enable gra… view at source ↗
Figure 2
Figure 2. Heatmap illustrating pairwise correlation coefficients between variables in the Traffic dataset. Most variable pairs exhibit positive correlations. 6 Discussions 6.1 Understanding Performance Degradation in Inter-Variable Models In this section, we qualitatively investigate why combining XicorAttention with models focusing on inter-variable relationships (e.g., iTransformer and TimeXer) sometimes leads to deteriorat… view at source ↗
Figure 3
Figure 3. Sensitivity analysis of forecasting performance (MSE) with respect to model dimension d ∈ {32, 64, 128, 256, 512} on ETTh1 and Electricity datasets (H = 96). The total attention dimension D is fixed at 512. {32, 64, 128, 256, 512} [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training time (in seconds) comparison across varying lookback lengths T ∈ {48, 96, 192, 336} on ETTh1 and Weather datasets with a fixed forecasting horizon H = 96. forecasting performance compared to existing Transformers on most datasets. In addition, integrating Xico…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages

  1. [1]

    SIAM Journal on Optimization10(3), 658–672 (2000)

    Best, M.J., Chakravarti, N., Ubhaya, V.A.: Minimizing separable convex functions subject to simple chain constraints. SIAM Journal on Optimization10(3), 658–672 (2000)

  2. [2]

    In: III, H.D., Singh, A

    Blondel, M., Teboul, O., Berthet, Q., Djolonga, J.: Fast differentiable sorting and ranking. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Con- ference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 950–959. PMLR (13–18 Jul 2020)

  3. [3]

    Chaos: An Inter- disciplinary Journal of Nonlinear Science25(9) (2015)

    Bradley, E., Kantz, H.: Nonlinear time-series analysis revisited. Chaos: An Inter- disciplinary Journal of Nonlinear Science25(9) (2015)

  4. [4]

    Journal of the American Statistical Association116(536), 2009–2022 (2021)

    Chatterjee, S.: A new coefficient of correlation. Journal of the American Statistical Association116(536), 2009–2022 (2021)

  5. [5]

    Scandinavian Journal of Statistics40(1), 21–41 (2013)

    Dette, H., Siburg, K.F., Stoimenov, P.A.: A copula-based non-parametric measure of regression dependence. Scandinavian Journal of Statistics40(1), 21–41 (2013)

  6. [6]

    In: International Conference on Learning Repre- sentations (2019) 16 D

    Grover, A., Wang, E., Zweig, A., Ermon, S.: Stochastic optimization of sorting net- works via continuous relaxations. In: International Conference on Learning Repre- sentations (2019) 16 D. Kimura et al

  7. [7]

    In: The 41st international ACM SIGIR conference on research & development in information retrieval

    Lai, G., Chang, W.C., Yang, Y., Liu, H.: Modeling long-and short-term tempo- ral patterns with deep neural networks. In: The 41st international ACM SIGIR conference on research & development in information retrieval. pp. 95–104 (2018)

  8. [8]

    Advances in neural information processing systems32(2019)

    Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.X., Yan, X.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems32(2019)

Show all 24 references
  1. [9]

    Biometrika 110(2), 283–299 (2023)

    Lin, Z., Han, F.: On boosting the power of chatterjee’s rank correlation. Biometrika 110(2), 283–299 (2023)

  2. [10]

    arXiv preprint arXiv:2310.06625 (2023)

    Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., Long, M.: itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 (2023)

  3. [11]

    In: International Conference on Learning Repre- sentations (2018)

    Mena, G., Belanger, D., Linderman, S., Snoek, J.: Learning latent permutations with gumbel-sinkhorn networks. In: International Conference on Learning Repre- sentations (2018)

  4. [12]

    In: ICASSP 2025 - 2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP)

    Nguyen, Q.M., Nguyen, L.M., Das, S.: Correlated attention in transformers for multivariate time series. In: ICASSP 2025 - 2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5 (2025). https://doi.org/10.1109/ICASSP49660.2025.10889540

  5. [13]

    In: The Eleventh International Conference on Learning Representations (2023)

    Nie, Y., Nguyen, N.H., Sinthong, P., Kalagnanam, J.: A time series is worth 64 words: Long-term forecasting with transformers. In: The Eleventh International Conference on Learning Representations (2023)

  6. [14]

    Advances in neural information processing sys- tems32(2019)

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing sys- tems32(2019)

  7. [15]

    In: International Conference on Machine Learning

    Prillo,S.,Eisenschlos,J.:Softsort:Acontinuousrelaxationfortheargsortoperator. In: International Conference on Machine Learning. pp. 7793–7802. PMLR (2020)

  8. [16]

    Nature344(6268), 734–741 (1990)

    Sugihara, G., May, R.M.: Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature344(6268), 734–741 (1990)

  9. [17]

    Advances in Neural Information Processing Systems (2017)

    Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems (2017)

  10. [18]

    Advances in Neural Information Processing Systems (2024)

    Wang, Y., Wu, H., Dong, J., Liu, Y., Qiu, Y., Zhang, H., Wang, J., Long, M.: Timexer: Empowering transformers for time series forecasting with exogenous vari- ables. Advances in Neural Information Processing Systems (2024)

  11. [19]

    url https://www.bgc-jena.mpg.de/wetter/

    Wetterstation: Weather. url https://www.bgc-jena.mpg.de/wetter/

  12. [20]

    In: International Conference on Learning Representations (2023)

    Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., Long, M.: Timesnet: Temporal 2d- variation modeling for general time series analysis. In: International Conference on Learning Representations (2023)

  13. [21]

    Advances in neural information processing systems34, 22419–22430 (2021)

    Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems34, 22419–22430 (2021)

  14. [22]

    In: The eleventh international conference on learning representations (2023)

    Zhang, Y., Yan, J.: Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In: The eleventh international conference on learning representations (2023)

  15. [23]

    In: Proceed- ings of the AAAI conference on artificial intelligence

    Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceed- ings of the AAAI conference on artificial intelligence. vol. 35, pp. 11106–11115 (2021)

  16. [24]

    In: International conference on machine learning

    Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: Fedformer: Frequency en- hanced decomposed transformer for long-term series forecasting. In: International conference on machine learning. pp. 27268–27286. PMLR (2022) XicorAttention: Attention with Nonlinear Correlation 1...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.