REVIEW 3 major objections 6 minor 24 references
XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Replacing dot-product attention with a rank-correlation score improves time-series forecasting up to 9.1 percent.
desk verdict A genuinely new attention scoring function built on a differentiable Chatterjee xi, but the empirical support is mixed and the 'sample' interpretation is shaky. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is Chatterjee's rank correlation coefficient $\xi_n(X,Y)$, defined as $1 - \frac{3\sum_{i=1}^{n-1}|r_{i+1}-r_i|}{n^2-1}$, where the pairs are sorted by $X$ and $r_i$ are the ranks of the corresponding $Y$ values. It converges to 0 if $X$ and $Y$ are independent and to 1 if $Y$ is a measurable function of $X$. To make it usable inside a Transformer, the paper approximates the sorting permutation with SoftSort and the ranking operation with FastSoftRank's projection onto the permutahedron, so gradients can flow back into the query and key projection matrices.
What would settle it
Train a Transformer model with XicorAttention on the same benchmarks while randomly permuting the coordinate positions of query and key vectors identically within each attention head before computing the $\xi$ score; if forecasting accuracy is unchanged, the mechanism merely rewards coordinate alignment and not genuine nonlinear dependence.
Extended reading notes
Core claim
The central claim is that computing attention weights with Chatterjee's $\xi$ correlation coefficient, rather than with the scaled dot product, lets Transformer models capture nonlinear query-key dependencies and thereby improves multivariate long-term time-series forecasting. Concretely, the paper replaces $QK^\top/\sqrt{d}$ in each attention head with $\xi_d(q,k)$, treating the $d$ coordinates of each query-key pair as paired samples. Because Chatterjee's $\xi$ is defined through sorting and ranking operations and is not differentiable, the paper builds a differentiable surrogate using SoftSort for the permutation step and FastSoftRank for the ranking step, with a straight-through trick to keep exact ordering in the forward pass. Experiments across ETT, Exchange, Weather, Electricity, and Traffic show that replacing the attention layer of PatchTST yields consistent gains, with the largest improvement being 9.12 percent on the Exchange dataset; gains are smaller or negative when the method is applied to models that emphasize inter-variable relationships, which the authors attribute to $\xi$'s weaker power on smooth, nearly linear dependencies.
Load-bearing premise
The paper assumes the coordinates of each query and key vector within a head behave like paired random samples from a joint distribution, so that a correlation statistic computed across those coordinates measures the true dependence between query and key; if the coordinates instead encode positional or structural information, the whole computation measures an artifact of coordinate ordering.
Editorial extensions
If this is right
- Any Transformer model whose attention layer is replaced by XicorAttention inherits a score that detects nonlinear and oscillatory dependencies, not just linear ones, at $O(n^2)$ complexity.
- For temporal-patching models like PatchTST, the mechanism gives consistent accuracy improvements across most tested datasets and horizons, so it can serve as a drop-in attention module for that model family.
- Head dimension $d$ acts as the sample size for the correlation estimate, and the paper's sensitivity analysis says accuracy stabilizes for $d \ge 128$, giving a concrete design rule for multi-head configurations.
- Avoiding the method for inter-variable attention on datasets with strong linear correlations, such as Traffic with 862 variates, prevents the observed performance degradation.
Reading between the lines
- The paper's diagnosis suggests a broader testable claim: attention mechanisms that measure only linear association underperform specifically on forecasting tasks where the relevant query-key dependence is nonlinear, and a correlation-based score should be most helpful when patched inputs or variates have oscillatory or threshold-like relationships.
- The interpretation of $\xi_d(q,k)$ as a dependence estimate rests on treating the $d$ coordinates of each attention head as exchangeable paired samples; for models where coordinates encode positional structure or patch order, the same computation could reward artifacts of coordinate arrangement rather than semantic relevance. An ablation that permutes coordinates within each head before computing
- Since the gain concentrates in PatchTST and degrades in iTransformer on Traffic, a natural next experiment is an adaptive scheme that selects between dot-product and $\xi$ per head or per dataset based on an estimated linearity of the dependency, rather than replacing attention uniformly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes XicorAttention, a replacement for the scaled dot-product in Transformer attention that uses a differentiable approximation of Chatterjee's rank correlation coefficient ξ. The authors employ SoftSort and FastSoftRank to make the sorting and ranking operations differentiable and integrate the mechanism into PatchTST, iTransformer, and TimeXer. The experiments cover seven multivariate time-series datasets with four forecasting horizons, and the abstract claims an improvement of up to approximately 9.1% over existing models. The paper reports that the integrated models achieve the best or second-best results on most datasets, but also documents substantial degradations, notably a 37.5% MSE increase when applied to iTransformer on Traffic.
Significance. The idea of using a rank-based nonlinear dependence measure inside attention is novel and could be relevant to time-series forecasting. The differentiable relaxation of sorting and ranking is technically sound and clearly presented. However, the statistical interpretation of computing ξ over the head-dimension coordinates as an estimate of dependence between query and key is not justified, and the empirical evidence for consistent improvement is mixed. If the statistical foundation can be repaired and the claims appropriately qualified, the work could be a useful empirical contribution. As it stands, the central claim at the level stated in the abstract is not supported.
major comments (3)
- [§3.1–3.2, Eq. (5)] The paper treats the d coordinate pairs (q_i, k_i) of the query and key vectors as i.i.d. samples for computing Chatterjee's ξ_n, stating in §3.2 that n and d are used interchangeably. This identification is invalid: q and k are deterministic learned projections of the same input, and the coordinates are not samples from any joint distribution. Consequently, ξ_d(q,k) is not a consistent estimator of a population dependence, and the claimed property of capturing nonlinear dependencies inherent in time series data does not follow from Chatterjee's theory. The authors should either provide a rigorous statistical model under which the coordinates are exchangeable samples, or reframe the method as a heuristic nonlinear similarity measure and adjust the motivation accordingly.
- [§5.2, Table 5; abstract] The abstract's claim that incorporating nonlinear correlation into the attention improves forecasting accuracy is not supported by the full set of results. On Traffic, replacing iTransformer's attention degrades MSE by 37.5% (from 0.422 to 0.570), and multiple other cells show negative enhancement, such as ETTh1 TimeXer MSE (-5.15%) and Electricity TimeXer MSE (-5.12%). The paper's acknowledgement of this behavior in Sections 5.2 and 6.1 should also be reflected in the abstract and in the 'consistently outperforms' statement in Section 5.1.
- [§6.2, Figure 3] The recommendation of a head dimension d≥128 is derived post hoc from the same sensitivity experiments used to motivate it. Since the paper identifies d with the sample size n for ξ, this recommendation conflates a heuristic architecture choice with a statistical claim about estimation accuracy. The authors should explicitly state that this is an empirical heuristic without a statistical justification.
minor comments (6)
- [§3.3] The phrase 'To archive this sorting operation' should be 'To achieve this sorting operation.'
- [§3.4] In the sentence explaining the permutahedron, 'the the elements' should be 'the elements.'
- [Figure 2 caption and §6.1] The Figure 2 caption says the heatmap shows the Traffic dataset, while the text in Section 6.1 says it shows the Weather dataset; please make these consistent.
- [Table 4] The header row of Table 4 is corrupted (e.g., 'Xicor+TimeXTimeXer', 'iT ransiTransformer'); please fix the formatting and column labels.
- [§3.3, Eq. (10)] The specific distance function d(x,y) used in SoftSort (e.g., L1 or L2) is not specified; please report the exact choice used in the experiments.
- [Throughout] The paper inconsistently abbreviates iTransformer as 'iTrans' in several places; please standardize the notation.
Circularity Check
No significant circularity: the forecasting improvements are measured empirically on held-out test sets, and the statistical properties of Chatterjee's xi are imported from external, non-self-cited work.
full rationale
The paper's central claim is that replacing scaled dot-product attention with a differentiable approximation of Chatterjee's xi improves multivariate time-series forecasting accuracy. This is an empirical architectural modification, not a derivation in which the output is equivalent to the input. The statistical properties of xi (e.g., xi = 0 under independence, xi = 1 for a measurable function) are cited from Chatterjee (2021) and Dette et al. (2013), which are external works with no author overlap. The differentiable sorting and ranking machinery is taken from Prillo & Eisenschlos (2020) and Blondel et al. (2020), also external. The claimed 9.1% improvement is a reported maximum across held-out benchmark evaluations, not a quantity constructed by definition from a fitted parameter. The identification of attention-head dimension d with the sample size n in Sections 3.1 and 3.2 is a modeling assumption that may be statistically questionable, but it is not circular: xi_n is not defined in terms of forecasting error, and forecasting error is not defined in terms of xi_n. The recommendation d >= 128 in Section 6.2 is a post-hoc observation from sensitivity experiments rather than a load-bearing prediction derived from the method's equations. There are no load-bearing self-citations, and no fitted input is renamed as a prediction. Accordingly, no circularity step can be exhibited by the paper's own equations or citation chain.
Assumptions & free parameters
free parameters (3)
- SoftSort temperature tau =
not specified
- FastSoftRank regularization epsilon =
not specified
- Attention head dimension d (sample size n for xi) =
recommended >= 128
assumptions (4)
- domain assumption The d coordinates of a query/key vector pair (q_i, k_i) are treated as an i.i.d. sample from a joint distribution.
- domain assumption The asymptotic limiting properties of Chatterjee's xi (0 iff independence, 1 iff deterministic relation) are assumed to hold approximately at finite head dimensions.
- standard math SoftSort and FastSoftRank provide faithful differentiable surrogates for sorting and ranking, with straight-through gradients that permit stable training.
- domain assumption Using xi as an attention score, followed by the model's existing normalization, still produces valid attention weights that can aggregate values.
Cite this review
Pith. "Pith review of XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation." pith.science (2026). https://pith.science/paper/WLR2CUEC
@misc{pith2026250602694,
author = {Pith},
title = {Pith review of: XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation},
year = {2026},
howpublished = {\url{https://pith.science/paper/WLR2CUEC}},
note = {Machine review of arXiv:2506.02694}
}
read the original abstract
Various Transformer-based models have been proposed for time series forecasting. These models leverage the self-attention mechanism to capture long-term temporal or variate dependencies in sequences. Existing methods can be divided into two approaches: (1) reducing computational cost of attention by making the calculations sparse, and (2) reshaping the input data to aggregate temporal features. However, existing attention mechanisms may not adequately capture inherent nonlinear dependencies present in time series data, leaving room for improvement. In this study, we propose a novel attention mechanism based on Chatterjee's rank correlation coefficient, which measures nonlinear dependencies between variables. Specifically, we replace the matrix multiplication in standard attention mechanisms with this rank coefficient to measure the query-key relationship. Since computing Chatterjee's correlation coefficient involves sorting and ranking operations, we introduce a differentiable approximation employing SoftSort and SoftRank. Our proposed mechanism, ``XicorAttention,'' integrates it into several state-of-the-art Transformer models. Experimental results on real-world datasets demonstrate that incorporating nonlinear correlation into the attention improves forecasting accuracy by up to approximately 9.1\% compared to existing models.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
SIAM Journal on Optimization10(3), 658–672 (2000)
Best, M.J., Chakravarti, N., Ubhaya, V.A.: Minimizing separable convex functions subject to simple chain constraints. SIAM Journal on Optimization10(3), 658–672 (2000)
work page 2000
-
[2]
Blondel, M., Teboul, O., Berthet, Q., Djolonga, J.: Fast differentiable sorting and ranking. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Con- ference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 950–959. PMLR (13–18 Jul 2020)
work page 2020
-
[3]
Chaos: An Inter- disciplinary Journal of Nonlinear Science25(9) (2015)
Bradley, E., Kantz, H.: Nonlinear time-series analysis revisited. Chaos: An Inter- disciplinary Journal of Nonlinear Science25(9) (2015)
work page 2015
-
[4]
Journal of the American Statistical Association116(536), 2009–2022 (2021)
Chatterjee, S.: A new coefficient of correlation. Journal of the American Statistical Association116(536), 2009–2022 (2021)
work page 2021
-
[5]
Scandinavian Journal of Statistics40(1), 21–41 (2013)
Dette, H., Siburg, K.F., Stoimenov, P.A.: A copula-based non-parametric measure of regression dependence. Scandinavian Journal of Statistics40(1), 21–41 (2013)
work page 2013
-
[6]
In: International Conference on Learning Repre- sentations (2019) 16 D
Grover, A., Wang, E., Zweig, A., Ermon, S.: Stochastic optimization of sorting net- works via continuous relaxations. In: International Conference on Learning Repre- sentations (2019) 16 D. Kimura et al
work page 2019
-
[7]
In: The 41st international ACM SIGIR conference on research & development in information retrieval
Lai, G., Chang, W.C., Yang, Y., Liu, H.: Modeling long-and short-term tempo- ral patterns with deep neural networks. In: The 41st international ACM SIGIR conference on research & development in information retrieval. pp. 95–104 (2018)
work page 2018
-
[8]
Advances in neural information processing systems32(2019)
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.X., Yan, X.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems32(2019)
work page 2019
Show all 24 references
-
[9]
Biometrika 110(2), 283–299 (2023)
Lin, Z., Han, F.: On boosting the power of chatterjee’s rank correlation. Biometrika 110(2), 283–299 (2023)
2023
-
[10]
arXiv preprint arXiv:2310.06625 (2023)
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., Long, M.: itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 (2023)
2023 arXiv
-
[11]
In: International Conference on Learning Repre- sentations (2018)
Mena, G., Belanger, D., Linderman, S., Snoek, J.: Learning latent permutations with gumbel-sinkhorn networks. In: International Conference on Learning Repre- sentations (2018)
2018
-
[12]
In: ICASSP 2025 - 2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP)
Nguyen, Q.M., Nguyen, L.M., Das, S.: Correlated attention in transformers for multivariate time series. In: ICASSP 2025 - 2025 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5 (2025). https://doi.org/10.1109/ICASSP49660.2025.10889540
2025
-
[13]
In: The Eleventh International Conference on Learning Representations (2023)
Nie, Y., Nguyen, N.H., Sinthong, P., Kalagnanam, J.: A time series is worth 64 words: Long-term forecasting with transformers. In: The Eleventh International Conference on Learning Representations (2023)
2023
-
[14]
Advances in neural information processing sys- tems32(2019)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing sys- tems32(2019)
2019
-
[15]
In: International Conference on Machine Learning
Prillo,S.,Eisenschlos,J.:Softsort:Acontinuousrelaxationfortheargsortoperator. In: International Conference on Machine Learning. pp. 7793–7802. PMLR (2020)
2020
-
[16]
Nature344(6268), 734–741 (1990)
Sugihara, G., May, R.M.: Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature344(6268), 734–741 (1990)
1990
-
[17]
Advances in Neural Information Processing Systems (2017)
Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems (2017)
2017
-
[18]
Advances in Neural Information Processing Systems (2024)
Wang, Y., Wu, H., Dong, J., Liu, Y., Qiu, Y., Zhang, H., Wang, J., Long, M.: Timexer: Empowering transformers for time series forecasting with exogenous vari- ables. Advances in Neural Information Processing Systems (2024)
2024
-
[19]
url https://www.bgc-jena.mpg.de/wetter/
Wetterstation: Weather. url https://www.bgc-jena.mpg.de/wetter/
-
[20]
In: International Conference on Learning Representations (2023)
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., Long, M.: Timesnet: Temporal 2d- variation modeling for general time series analysis. In: International Conference on Learning Representations (2023)
2023
-
[21]
Advances in neural information processing systems34, 22419–22430 (2021)
Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems34, 22419–22430 (2021)
2021
-
[22]
In: The eleventh international conference on learning representations (2023)
Zhang, Y., Yan, J.: Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In: The eleventh international conference on learning representations (2023)
2023
-
[23]
In: Proceed- ings of the AAAI conference on artificial intelligence
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceed- ings of the AAAI conference on artificial intelligence. vol. 35, pp. 11106–11115 (2021)
2021
-
[24]
In: International conference on machine learning
Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: Fedformer: Frequency en- hanced decomposed transformer for long-term series forecasting. In: International conference on machine learning. pp. 27268–27286. PMLR (2022) XicorAttention: Attention with Nonlinear Correlation 1...
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.