Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Standard transformers fail at recommendation because long-tail, heterogeneous feature data collapses token embeddings into a low-rank space; SpecFormer breaks the resulting vicious cycle by softly rebalancing the singular-value spectrum.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:14 UTC pith:LXOSLF2H

load-bearing objection Genuinely novel spectral attention recipe with a strong online A/B signal, but the theoretical 'vicious cycle' is not established. the 4 major comments →

arxiv 2607.24025 v1 pith:LXOSLF2H submitted 2026-07-27 cs.IR cs.LG

SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation

Yu Cui , Yi Xu , Jiahao Wang , Hao Zhang , Yu Zhang , Xiaoyi Zeng , Can Wang , Jinxin Hu
show 1 more author
Jiawei Chen
This is my paper
classification cs.IR cs.LG
keywords recommender systemsCTR predictionTransformerembedding collapseattention collapsespectral collapsesingular value decompositioneffective rank
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that standard self-attention underperforms in Click-Through-Rate prediction because the two signatures of recommendation data — heterogeneous feature fields and a long-tail of feature frequencies — drive the token embeddings into a spectral collapse in which a handful of singular values dominate. That collapse, the paper argues, triggers a vicious cycle: collapsed embeddings make the attention map nearly rank-1 in the forward pass, and that degenerate attention acts as a low-pass filter in the backward pass, starving the minor spectral components of gradient and collapsing the embeddings further. To break the cycle, the paper proposes SpecFormer, a spectral-aware Transformer that flattens the singular-value spectrum of the embeddings with a learnable power-law transform, computes attention from the softened embeddings, and adds a spectral residual position encoding. The paper reports that SpecFormer outperforms state-of-the-art feature-interaction baselines on one industrial and two public CTR datasets, and that stacking SpecFormer layers raises the attention's effective rank rather than destroying it. If correct, the result matters because it shows that Transformers can be made depth-scalable for large-scale recommendation without abandoning the attention mechanism; the paper's online A/B test (CTR +1.34%, orders +16.72%) indicates the effect survives production traffic.

Core claim

The paper's central claim is that a 'vicious cycle' of spectral collapse explains why standard self-attention collapses at shallow depth in recommender systems. In the forward direction (Proposition III.1), when the token embedding matrix H has SVD H = UΣV^T with σ_k ≤ εσ_1 for k≥2, the pre-softmax attention score S = H W_q W_k^T H^T is dominated by the rank-1 term σ_1^2 M_11 u_1 u_1^T, and softmax exponentially amplifies that dominance, yielding a nearly rank-1 attention map. In the backward direction (Proposition III.2), that degenerate attention map acts as a low-pass filter: projected onto the singular basis U, it preserves the primary component u_1 but attenuates the minor components u_

What carries the argument

The central object is the singular-value spectrum of the token embedding matrix H = UΣV^T and its concentration measured by the cumulative singular value ratio ρ(k). The load-bearing identity is the spectrum-softened attention score S = U (Σ^* V^T W_q W_k^T V Σ^*) U^T, where Σ^* = diag(σ_i^τ) with learnable τ ∈ (0,1). The machinery has three parts: (i) Learnable Spectral Softening, which flattens the spectrum by raising each singular value to a learnable power, redistributing energy from dominant to tail components; (ii) Spectrum-softened Attention, which builds Query and Key from the softened embeddings and the Value from the un-softened embeddings, so that no single spectral component domi

Load-bearing premise

The backward-propagation half of the vicious cycle depends on the premise, borrowed from language-model analysis and not re-derived for recommendation data, that a degenerate attention matrix acts as a low-pass filter that strongly attenuates the minor spectral components of the gradient (the attenuation factor ξ_k is far smaller than ξ_1); if that premise fails for recommendation-scale attention matrices, the cycle is not established even though SpecFormer may still improve

What would settle it

Measure, on a trained attention-based Transformer baseline (the kind used in the paper's comparisons) on a public CTR dataset, the backward attenuation factor ξ_k = ||A^T u_k|| / ||u_k|| for k=2..r using the attention matrix at each layer. If ξ_k is not consistently far smaller than ξ_1 (i.e., not ξ_k << ξ_1), then the claimed backward-propagation amplification is contradicted by data. A second, complementary check: train SpecFormer with spectral softening disabled but the residual position encodings and two-stage protocol intact; if performance on the public datasets stays close to the full m

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Depth scaling becomes possible for Transformer-based CTR models: stacking SpecFormer layers increases both the attention's fractional effective rank and AUC, whereas standard attention degrades after a few layers.
  • Long-tail features receive meaningful gradient updates: softening revives minor spectral directions that would otherwise be starved, which the paper identifies as the mechanism behind the accuracy gains.
  • Spectral diagnostics predict failure: the cumulative singular value ratio of embeddings and the fractional effective rank of attention can serve as early indicators of when a Transformer architecture will collapse on a recommendation dataset.
  • The design is production-viable: in an online A/B test the paper reports CTR +1.34%, CVR +15.97%, and orders +16.72% at a 5 ms latency increase, with a 35% training-efficiency gain from the in-batch SVD and system optimizations.
  • The ablation that removes Spectral Softening causes a large AUC drop (e.g., from 0.7587 to 0.6555 on one public dataset), which the paper interprets as direct evidence that dynamic spectral flattening—not merely extra parameters—is what prevents collapse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The vicious-cycle mechanism likely extends beyond CTR: any sparse, long-tailed, high-cardinality categorical learning task (e.g., sequential recommendation, tabular learning) should exhibit the same forward/backward collapse, and the same diagnostics could be used to test it.
  • The paper imports the 'attention as low-pass filter' property from language-model analysis rather than proving it for recommendation; a direct measurement of the attenuation factor ξ_k on recommendation attention matrices would settle whether the backward half of the cycle is as strong as claimed.
  • The spectral-softening operator could be treated as a portable module and inserted into non-attention feature-interaction models (e.g., MLP-based mixers) to test whether flattening the singular-value spectrum alone accounts for the gains, which the paper does not explore.
  • The optimal residual weights (α=0.001 vs γ=0.8) differ by three orders of magnitude; a natural follow-up is to normalize the spectral bias by its spectral norm so the trade-off becomes scale-free and easier to tune in other deployments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that standard self-attention underperforms in CTR recommendation because heterogeneous, long-tail input embeddings undergo spectral collapse, which then triggers a vicious cycle: collapsed embeddings produce low-rank attention maps (forward propagation), and those attention maps act as low-pass filters that starve minor spectral components of gradients (backward propagation). To break this cycle, the authors propose SpecFormer, which applies learnable power-law softening to the singular values of the input embeddings, computes attention in the softened spectral space, and adds a Taylor-expanded spectral residual position encoding plus a spatial residual term. The paper reports offline gains on one industrial and two public datasets, online A/B improvements, and depth-scaling behavior.

Significance. If the mechanism were fully established, the paper would make an important conceptual contribution by explaining why Transformer depth fails in CTR prediction and by proposing a spectral remedy with industrial validation. Strengths include a real deployed system, online A/B results, a public code release, and a scaling study that compares standard attention with the proposed architecture. The paper also provides a useful set of ablations. However, the core theoretical claim—the forward/backward vicious cycle—relies on bounds and imported properties that are not justified for the actual softmax attention objects in recommendation. The empirical comparison also introduces a training-protocol difference between SpecFormer and baselines. These issues are load-bearing for the paper's central narrative, though they appear addressable in revision.

major comments (4)
  1. [Section III, Proposition III.1, Eq. (11)] The bound on R_ij is stated as O(εσ1^2), but R_ij is a sum over r^2−1 terms, each only O(εσ1^2) when at least one index is not 1. The correct entrywise bound is O(r ε σ1^2) unless M is sparse or r is constant. With r equal to the number of tokens (dozens/hundreds), the residual can be comparable to the dominant term even for small ε. Moreover, the step from 'pre-softmax scores dominated by a rank-1 component' to 'softmax attention is low-rank' is not automatic: softmax of a rank-1 matrix is not generally low-rank, especially when u1 contains mixed signs. Please provide a direct bound on ||softmax(S)−softmax(S1)|| or on the attention effective rank.
  2. [Section III, Proposition III.2, Eqs. (13)-(15)] The backward-propagation claim rests on the assertion that A1 acts as a low-pass filter with attenuation factors ξ_k ≪ ξ_1. This property is imported from [38], which studies rank collapse in deep pure-attention stacks, not a single softmax attention layer on categorical, heterogeneous embeddings. Softmax of S1 = σ1^2 M11 u1 u1^T can have high-frequency entries when u1 is not nonnegative, so ξ_k need not be small. The attenuation factors are not derived and not measured. Without this, the gradient-starvation half of the vicious cycle is not established. Either prove the low-pass property for the actual A1 in CTR settings or measure ξ_k empirically from trained models.
  3. [Section V-A, Table III] SpecFormer is trained with a two-stage protocol (5% warm-up on the spatial residual term, then 95% main CTR training), while every baseline is trained with a single-stage protocol on 100% of the data. This is a training-protocol confound: the reported AUC/GAUC/Logloss gains could stem from the warm-up schedule or the data split rather than the spectral mechanism. To support the state-of-the-art claim, run baselines with the same warm-up stage (or ablate the warm-up from SpecFormer) and report whether the gains persist.
  4. [Section V-D and Eqs. (17), (9)] The scaling result that SpecFormer's fractional effective rank increases with depth may be partly mechanical: Eq. (17) explicitly flattens the singular-value spectrum used to compute Q and K, and ferank (Eq. 9) measures the entropy of the attention singular values. Thus an increasing ferank does not by itself establish richer feature interaction learning. Report the ferank of a no-softening ablation at matched depths, and show that the AUC gains are not explainable solely by spectral flattening.
minor comments (5)
  1. [Section V-E and Figure 6] The text says α values are {0.0001, 0.001, 0.01}, while the figure x-axis shows {0.0005, 0.001, 0.002}. Please align the reported hyperparameter grid.
  2. [Author block] The author affiliations contain ordinal typos: '4rd', '5rd', '8rd'. Also, the use of future-dated venues and arXiv IDs (e.g., [6] WWW'26, [11] CIKM'25, [15] arXiv:2604.17878) should be checked before publication.
  3. [Table III, FEDIN row] FEDIN's Logloss on Criteo (0.9568) and Avazu (0.4256) is far outside the range of other baselines; please verify the implementation or hyperparameter settings for this baseline.
  4. [Section V-C, Table IV] Ablation (a) 'w/o Spectral Softening' on Industrial shows only a small AUC drop (0.7611→0.7576) and an improved Logloss (0.1119→0.1100), which is inconsistent with the text's 'catastrophic performance decline'. Report variance or significance tests.
  5. [Section IV-A, Eq. (17)] For singular values below 1, the power-law operation σ_i^τ with τ<1 increases the value, so the statement that it 'compresses large singular values more aggressively than small ones' describes a relative, not absolute, effect. Please clarify the intended regime and the effect of the τ=Sigmoid(a) initialization.

Circularity Check

0 steps flagged

No material circularity; theoretical chain is conditional and externally cited; minor self-citations are not load-bearing.

full rationale

The derivation chain in Section III is conditional rather than circular. Proposition III.1 explicitly assumes the spectral-collapse premise (sigma_k <= epsilon*sigma_1 for k>=2) and algebraically derives dominance of the principal component in S = U*Sigma*M*Sigma*U^T. The conclusion is not used to define the premise; it is a mathematical implication from a stated assumption. Proposition III.2 then assumes the degenerate attention produced by that conditional result and imports the low-pass-filter property from external prior work [38] (Dong et al., with no author overlap). The gradient-starvation conclusion is therefore a chain of conditional arguments, not a reduction of the conclusion to an input by construction. The paper's scaling evidence is empirical (Tables III/V, Fig. 5) and is not a fitted parameter renamed as a prediction: the learnable tau in Eq. (17) is trained, but the reported AUC/CTR gains and the online A/B test are external outcomes not equal to the ferank objective. Although Eq. (17) flattens the embedding singular-value spectrum and ferank measures spectral flatness, ferank is computed on the attention matrix, not directly on the embedding spectrum, and the headline scaling claim also includes performance gains that are not forced by the spectral design. There are self-citations by overlapping authors, notably [13] for recommendation-data spectral collapse and [32] for the cumulative-singular-value metric, but they are not load-bearing: the paper supplies its own empirical evidence (Figs. 2-3), the theorem explicitly assumes the collapse premise rather than deriving it from the citation, and the metric is standard. No uniqueness theorem or ansatz is imported from the authors' own prior work; the low-pass-filter assumption comes from independent prior work. Thus no circular step meets the standard of exhibiting Eq. X = Eq. Y by construction or a fitted parameter called a prediction. The minor self-citations are acknowledged but do not affect the central derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

No new physical entities are introduced. The theoretical load is carried by the severe-collapse premise, the low-pass attention premise, the differentiability of the SVD layer, and the data-driven collapse premise from [13]; the method also introduces learned/tuned parameters τ, β, α, γ, and P. This ledger measures the paper's dependence on assumptions it did not independently prove.

free parameters (5)
  • a (learnable softening scalar; τ = sigmoid(a)) = τ starts at 0.5; final trained value not reported
    Eq. (17): controls the degree of spectral flattening in the Learnable Spectral Softening module.
  • β = [β0, β1, β2] = not reported
    Eq. (24): Taylor-expansion-like coefficients in the spectral residual bias; learned during training.
  • α = ≈0.001 on Industrial dataset
    Eq. (27): strength of the spectral residual bias; tuned per dataset via hyperparameter search.
  • γ = ≈0.8 on Industrial dataset
    Eq. (27): strength of the spatial residual bias; tuned per dataset via hyperparameter search.
  • P(l) spectral interaction matrix = learned r×r matrix; not reported
    Eq. (22): parameterizes the spectral residual position encoding; if r ≈ L, this is a large additional parameter set.
axioms (6)
  • domain assumption Severe spectral collapse premise: σ_k ≤ εσ_1 for all k≥2 at the input/layer embeddings
    Used as the premise of Proposition III.1; empirically supported by Fig. 3 but not derived from first principles in this paper.
  • domain assumption Self-attention matrix acts as a low-pass filter in backward propagation (attenuation factor ξ_k << ξ_1)
    Proposition III.2 relies on this property, citing [38]; it is not verified for recommendation attention matrices in this paper.
  • domain assumption SVD plus power-law softening remains differentiable and numerically stable during training
    Eqs. (16)-(18) require per-layer SVD; the paper does not analyze gradient stability or conditioning when singular values approach zero.
  • domain assumption Data heterogeneity and long-tail distributions induce the initial embedding collapse
    Section I and Section VI rely on [13] (with overlapping authors) and on Figures 2-3; this is the trigger of the vicious cycle.
  • domain assumption Weight matrices Wq, Wk have bounded norms so M = V^T Wq Wk^T V is small
    Used in the proof of Proposition III.1; standard regularization is mentioned but no explicit bound is derived.
  • standard math Softmax of a rank-1 dominant matrix remains effectively low-rank, and softmax is Lipschitz continuous
    Used in Propositions III.1 and III.2; standard results [39] are cited.

pith-pipeline@v1.3.0-alltime-deepseek · 19812 in / 19882 out tokens · 183709 ms · 2026-07-31T23:14:32.024541+00:00 · methodology

0 comments
read the original abstract

Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.

Figures

Figures reproduced from arXiv: 2607.24025 by Can Wang, Hao Zhang, Jiahao Wang, Jiawei Chen, Jinxin Hu, Xiaoyi Zeng, Yi Xu, Yu Cui, Yu Zhang.

Figure 1
Figure 1. Figure 1: The comparison on fractional effective rank of attention [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) Visualization of the token semantic space distributions of Qwen3 and OneTrans, indicating evident heterogeneity [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Token frequency exhibits a severe long-tail dis [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The architecture of SpecFormer. Each SpecFormer attention layer consists of three components: (1) Learnable Spectral [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of AUC gains and attention effective ranks [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Hyperparameter sensitivity analysis of SpecFormer. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 16 linked inside Pith

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186

    2019

  3. [3]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  4. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  5. [5]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  6. [6]

    Onetrans: Unified feature interaction and sequence modeling with one transformer in industrial recommender,

    Z. Zhang, H. Pei, J. Guo, T. Wang, Y . Feng, H. Sun, S. Liu, and A. Sun, “Onetrans: Unified feature interaction and sequence modeling with one transformer in industrial recommender,” inProceedings of the ACM Web Conference 2026, 2026, pp. 8162–8170

    2026

  7. [7]

    Final: Factorized interaction layer for ctr prediction,

    J. Zhu, Q. Jia, G. Cai, Q. Dai, J. Li, Z. Dong, R. Tang, and R. Zhang, “Final: Factorized interaction layer for ctr prediction,” inProceedings of the 46th International ACM SIGIR conference on research and development in information retrieval, 2023, pp. 2006–2010

    2023

  8. [8]

    Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,

    F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang, “Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,” inProceedings of the 28th ACM international confer- ence on information and knowledge management, 2019, pp. 1441–1450

    2019

  9. [9]

    Self-attentive sequential recommenda- tion,

    W.-C. Kang and J. McAuley, “Self-attentive sequential recommenda- tion,” in2018 IEEE international conference on data mining (ICDM). IEEE, 2018, pp. 197–206

    2018

  10. [10]

    Autoint: Automatic feature interaction learning via self-attentive neural networks,

    W. Song, C. Shi, Z. Xiao, Z. Duan, Y . Xu, M. Zhang, and J. Tang, “Autoint: Automatic feature interaction learning via self-attentive neural networks,” inProceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1161–1170

    2019

  11. [11]

    Rankmixer: Scaling up ranking models in industrial recommenders,

    J. Zhu, Z. Fan, X. Zhu, Y . Jiang, H. Wang, X. Han, H. Ding, X. Wang, W. Zhao, Z. Gonget al., “Rankmixer: Scaling up ranking models in industrial recommenders,” inProceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, pp. 6309–6316

    2025

  12. [12]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    How do recommendation models amplify popularity bias? an analysis from the spectral perspective,

    S. Lin, C. Gao, J. Chen, S. Zhou, B. Hu, Y . Feng, C. Chen, and C. Wang, “How do recommendation models amplify popularity bias? an analysis from the spectral perspective,” inProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, 2025, pp. 659–668

    2025

  14. [14]

    On the embedding collapse when scaling up recommendation models,

    X. Guo, J. Pan, X. Wang, B. Chen, J. Jiang, and M. Long, “On the embedding collapse when scaling up recommendation models,”arXiv preprint arXiv:2310.04400, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Rankup: Towards high-rank representa- tions for large scale advertising recommender systems,

    J. Chen, S. Zhang, B. Hu, C. Zhou, J. Pan, G. Xue, W. Ning, G. Weng, W. Zheng, S. Liuet al., “Rankup: Towards high-rank representa- tions for large scale advertising recommender systems,”arXiv preprint arXiv:2604.17878, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    From scaling to structured expressivity: Rethinking transformers for ctr prediction,

    B. Yan, Y . Lei, Z. Zeng, D. Wang, K. Lin, P. Wang, J. Xu, and B. Zheng, “From scaling to structured expressivity: Rethinking transformers for ctr prediction,”arXiv preprint arXiv:2511.12081, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Are id embeddings nec- essary? whitening pre-trained text embeddings for effective sequential recommendation,

    L. Zhang, X. Zhou, Z. Zeng, and Z. Shen, “Are id embeddings nec- essary? whitening pre-trained text embeddings for effective sequential recommendation,” in2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024, pp. 530–543

    2024

  18. [18]

    Fedin: Frequency-enhanced deep interest network for click-through rate pre- diction,

    Z. Dai, J. Wang, J. Pan, D. Liu, L. Xiao, and S.-T. Xia, “Fedin: Frequency-enhanced deep interest network for click-through rate pre- diction,”arXiv preprint arXiv:2605.01726, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    A convolutional click prediction model,

    Q. Liu, F. Yu, S. Wu, and L. Wang, “A convolutional click prediction model,” inProceedings of the 24th ACM international on conference on information and knowledge management, 2015, pp. 1743–1746

    2015

  20. [20]

    Fibinet: combining feature impor- tance and bilinear feature interaction for click-through rate prediction,

    T. Huang, Z. Zhang, and J. Zhang, “Fibinet: combining feature impor- tance and bilinear feature interaction for click-through rate prediction,” inProceedings of the 13th ACM conference on recommender systems, 2019, pp. 169–177

    2019

  21. [21]

    Field-weighted factorization machines for click-through rate prediction in display advertising,

    J. Pan, J. Xu, A. L. Ruiz, W. Zhao, S. Pan, Y . Sun, and Q. Lu, “Field-weighted factorization machines for click-through rate prediction in display advertising,” inProceedings of the 2018 world wide web conference, 2018, pp. 1349–1357

    2018

  22. [22]

    Fm2: Field-matrixed factor- ization machines for recommender systems,

    Y . Sun, J. Pan, A. Zhang, and A. Flores, “Fm2: Field-matrixed factor- ization machines for recommender systems,” inProceedings of the web conference 2021, 2021, pp. 2828–2837

    2021

  23. [23]

    Deepfm: a factorization- machine based neural network for ctr prediction,

    H. Guo, R. Tang, Y . Ye, Z. Li, and X. He, “Deepfm: a factorization- machine based neural network for ctr prediction,”arXiv preprint arXiv:1703.04247, 2017

    Pith/arXiv arXiv 2017

  24. [24]

    Fibinet++: Reducing model size by low rank feature interaction layer for ctr prediction,

    P. Zhang, Z. Zheng, and J. Zhang, “Fibinet++: Reducing model size by low rank feature interaction layer for ctr prediction,” inProceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 4425–4429

    2023

  25. [25]

    Factorization machines,

    S. Rendle, “Factorization machines,” in2010 IEEE International con- ference on data mining. IEEE, 2010, pp. 995–1000

    2010

  26. [26]

    Attentional factorization machines: Learning the weight of feature interactions via attention networks,

    J. Xiao, H. Ye, X. He, H. Zhang, F. Wu, and T.-S. Chua, “Attentional factorization machines: Learning the weight of feature interactions via attention networks,”arXiv preprint arXiv:1708.04617, 2017

    Pith/arXiv arXiv 2017

  27. [27]

    Wide & deep learning for recommender systems,

    H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispiret al., “Wide & deep learning for recommender systems,” inProceedings of the 1st workshop on deep learning for recommender systems, 2016, pp. 7–10

    2016

  28. [28]

    Hiformer: Heterogeneous feature interactions learning with trans- formers for recommender systems,

    H. Gui, R. Wang, K. Yin, L. Jin, M. Kula, T. Xu, L. Hong, and E. H. Chi, “Hiformer: Heterogeneous feature interactions learning with trans- formers for recommender systems,”arXiv preprint arXiv:2311.05884, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  30. [30]

    Svd approach to data unfolding,

    A. Hoecker and V . Kartvelishvili, “Svd approach to data unfolding,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 372, no. 3, pp. 469–481, 1996

    1996

  31. [31]

    Towards mitigat- ing dimensional collapse of representations in collaborative filtering,

    H. Chen, V . Lai, H. Jin, Z. Jiang, M. Das, and X. Hu, “Towards mitigat- ing dimensional collapse of representations in collaborative filtering,” in Proceedings of the 17th ACM international conference on web search and data mining, 2024, pp. 106–115

    2024

  32. [32]

    Spectran: Spectral-aware transformer-based adapter for llm-enhanced sequential recommendation,

    Y . Cui, F. Liu, Z. Wang, C. Zhang, J. Wang, C. Wang, and J. Chen, “Spectran: Spectral-aware transformer-based adapter for llm-enhanced sequential recommendation,”arXiv preprint arXiv:2601.21986, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    The effective rank: A measure of effective dimensionality,

    O. Roy and M. Vetterli, “The effective rank: A measure of effective dimensionality,” in2007 15th European signal processing conference. IEEE, 2007, pp. 606–610

    2007

  34. [34]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,”The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948

    1948

  35. [35]

    Signal propagation in transformers: Theoretical perspectives and the role of rank collapse,

    L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Luc- chi, “Signal propagation in transformers: Theoretical perspectives and the role of rank collapse,”Advances in Neural Information Processing Systems, vol. 35, pp. 27 198–27 211, 2022

    2022

  36. [36]

    Diff-erank: A novel rank- based metric for evaluating large language models,

    L. Wei, Z. Tan, C. Li, J. Wang, and W. Huang, “Diff-erank: A novel rank- based metric for evaluating large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 39 501–39 521, 2024

    2024

  37. [37]

    Dcn v2: Improved deep & cross network and practical lessons for web- scale learning to rank systems,

    R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi, “Dcn v2: Improved deep & cross network and practical lessons for web- scale learning to rank systems,” inProceedings of the web conference 2021, 2021, pp. 1785–1797

    2021

  38. [38]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth,

    Y . Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” in International conference on machine learning. PMLR, 2021, pp. 2793– 2803

    2021

  39. [39]

    On the properties of the softmax function with application in game theory and reinforcement learning,

    B. Gao and L. Pavel, “On the properties of the softmax function with application in game theory and reinforcement learning,”arXiv preprint arXiv:1704.00805, 2017

    Pith/arXiv arXiv 2017

  40. [40]

    Masknet: Introducing feature-wise multiplication to ctr ranking models by instance-guided mask,

    Z. Wang, Q. She, and J. Zhang, “Masknet: Introducing feature-wise multiplication to ctr ranking models by instance-guided mask,”arXiv preprint arXiv:2102.07619, 2021

    Pith/arXiv arXiv 2021

  41. [41]

    Towards deeper, lighter and interpretable cross network for ctr prediction,

    F. Wang, H. Gu, D. Li, T. Lu, P. Zhang, and N. Gu, “Towards deeper, lighter and interpretable cross network for ctr prediction,” in Proceedings of the 32nd ACM international conference on information and knowledge management, 2023, pp. 2523–2533

    2023

  42. [42]

    Deep interest network for click-through rate prediction,

    G. Zhou, X. Zhu, C. Song, Y . Fan, H. Zhu, X. Ma, Y . Yan, J. Jin, H. Li, and K. Gai, “Deep interest network for click-through rate prediction,” inProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 1059–1068

    2018

  43. [43]

    Adaptive subgradient methods for online learning and stochastic optimization

    J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradient methods for online learning and stochastic optimization.”Journal of machine learning research, vol. 12, no. 7, 2011

    2011

  44. [44]

    A generic learning framework for sequential recommendation with distribution shifts,

    Z. Yang, X. He, J. Zhang, J. Wu, X. Xin, J. Chen, and X. Wang, “A generic learning framework for sequential recommendation with distribution shifts,” inProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 331–340

    2023

  45. [45]

    Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction,

    Q. Pi, G. Zhou, Y . Zhang, Z. Wang, L. Ren, Y . Fan, X. Zhu, and K. Gai, “Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction,” inProceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020, pp. 2685–2692

    2020

  46. [46]

    Tallrec: An effective and efficient tuning framework to align large language model with recommendation,

    K. Bao, J. Zhang, Y . Zhang, W. Wang, F. Feng, and X. He, “Tallrec: An effective and efficient tuning framework to align large language model with recommendation,” inProceedings of the 17th ACM Conference on Recommender Systems, 2023, pp. 1007–1014

    2023

  47. [47]

    Hllm: Enhancing sequential recommendations via hierarchical large language models for item and user modeling,

    J. Chen, L. Chi, B. Peng, and Z. Yuan, “Hllm: Enhancing sequential recommendations via hierarchical large language models for item and user modeling,”arXiv preprint arXiv:2409.12740, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation,

    J. Lin, R. Shan, C. Zhu, K. Du, B. Chen, S. Quan, R. Tang, Y . Yu, and W. Zhang, “Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation,” inProceedings of the ACM on Web Conference 2024, 2024, pp. 3497–3508

    2024

  49. [49]

    Breaking the length barrier: Llm-enhanced ctr prediction in long textual user behaviors,

    B. Geng, Z. Huan, X. Zhang, Y . He, L. Zhang, F. Yuan, J. Zhou, and L. Mo, “Breaking the length barrier: Llm-enhanced ctr prediction in long textual user behaviors,” inProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 2311–2315

    2024

  50. [50]

    Ctrl: Connect collaborative and language model for ctr prediction,

    X. Li, B. Chen, L. Hou, and R. Tang, “Ctrl: Connect collaborative and language model for ctr prediction,”ACM Transactions on Recommender Systems, vol. 4, no. 2, pp. 1–23, 2025

    2025

  51. [51]

    Towards open-world recommendation with knowledge augmentation from large language models,

    Y . Xi, W. Liu, J. Lin, X. Cai, H. Zhu, J. Zhu, B. Chen, R. Tang, W. Zhang, and Y . Yu, “Towards open-world recommendation with knowledge augmentation from large language models,” inProceedings of the 18th ACM Conference on Recommender Systems, 2024, pp. 12– 22

    2024

  52. [52]

    Large language models enhanced collaborative filtering,

    Z. Sun, Z. Si, X. Zang, K. Zheng, Y . Song, X. Zhang, and J. Xu, “Large language models enhanced collaborative filtering,” inProceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 2178–2188

    2024

  53. [53]

    Field matters: A lightweight llm-enhanced method for ctr prediction,

    Y . Cui, F. Liu, J. Chen, X. Lou, C. Zhang, J. Wang, Y . Sun, X. Yang, and C. Wang, “Field matters: A lightweight llm-enhanced method for ctr prediction,” inProceedings of the ACM Web Conference 2026, 2026, pp. 6365–6376

    2026

  54. [54]

    Representation degeneration problem in training natural language generation models,

    J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y . Liu, “Representation degeneration problem in training natural language generation models,” arXiv preprint arXiv:1907.12009, 2019

    Pith/arXiv arXiv 1907

  55. [55]

    Understanding dimen- sional collapse in contrastive self-supervised learning,

    L. Jing, P. Vincent, Y . LeCun, and Y . Tian, “Understanding dimen- sional collapse in contrastive self-supervised learning,”arXiv preprint arXiv:2110.09348, 2021

    Pith/arXiv arXiv 2021

  56. [56]

    On feature decorrelation in self-supervised learning,

    T. Hua, W. Wang, Z. Xue, S. Ren, Y . Wang, and H. Zhao, “On feature decorrelation in self-supervised learning,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9598– 9608

    2021

  57. [57]

    Alphafuse: Learn id embeddings for sequential recommendation in null space of language embeddings,

    G. Hu, A. Zhang, S. Liu, Z. Cai, X. Yang, and X. Wang, “Alphafuse: Learn id embeddings for sequential recommendation in null space of language embeddings,” inProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025, pp. 1614–1623

    2025

  58. [58]

    Low-rank bottleneck in multi-head attention models,

    S. Bhojanapalli, C. Yun, A. S. Rawat, S. Reddi, and S. Kumar, “Low-rank bottleneck in multi-head attention models,” inInternational conference on machine learning. PMLR, 2020, pp. 864–873

    2020