Pith. sign in

REVIEW 5 major objections 4 minor 58 references

SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation

T0 review · 5 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpecFormer is a spectral-aware Transformer that flattens the singular-value spectrum of embeddings to prevent embedding/attention collapse, outperforming baselines on CTR benchmarks and scaling with layer depth.

desk verdict SpecFormer is a genuinely novel spectral-softening attention for CTR that shows real gains, but the theory is asserted rather than proved and the experiments need cleanup — worth refereeing, not accepting as-is. read the letter →

arxiv 2607.24025 v2 pith:LXOSLF2H submitted 2026-07-27 cs.IR cs.LG

Yu Cui , Yi Xu , Jiahao Wang , Hao Zhang , Yu Zhang , Xiaoyi Zeng , Can Wang , Jinxin Hu
show 1 more author
Jiawei Chen
This is my paper
classification cs.IRcs.LG
keywords recommendationattentionspecformercollapsespectralembeddingmodelperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Recommender systems predict clicks by combining many features—user IDs, item IDs, context—and Transformer models try to learn interactions between features. The authors observe that in recommendation data, a few very frequent features dominate, so the embedding matrix has a very uneven distribution of singular values (a measure of how many independent directions the data uses). They call this spectral collapse. When standard attention is applied, the attention matrix also collapses to a low-rank pattern, and gradients during training concentrate on the dominant directions, making the collapse worse as layers are added.

SpecFormer changes the attention computation. It first computes the SVD of the token embeddings, then applies a learnable power-law softening to the singular values, compressing the large ones more than the small ones. This flattens the spectrum. Attention is then computed in this softened space, using the original embeddings as values to avoid information loss. Two residual bias terms are added: one built from a Taylor expansion of normalized singular values to preserve the dominant signals, and one from the original embedding space for stability.

The paper reports consistent AUC improvements over strong baselines on Criteo, Avazu, and a private industrial dataset, plus an online A/B test with +1.34% CTR and +16.72% orders at 5ms extra latency. They also show that stacking SpecFormer layers increases, rather than decreases, the effective rank of attention.

Extended reading notes

Core claim

The authors claim that 'the heterogeneity and long-tail distribution characteristics of recommendation data catalyze a vicious cycle of spectral collapse for both feature embeddings and attention maps' (Section I) and that SpecFormer 'breaks the vicious cycle' via learnable spectral softening, spectrum-softened attention, and spectral residual position encoding, so that 'stacking SpecFormer layers actively improves the attention effective rank and recommendation performance' (Abstract/Section V-D). If true, standard self-attention in CTR models collapses because the embedding spectrum is dominated by a few singular values, and SpecFormer's spectrum-flattening design prevents that collapse and enables depth scaling.

Load-bearing premise

The load-bearing premise is Proposition III.1's assumption that the token embedding matrix already satisfies σ_k ≤ εσ_1 for all k≥2, i.e. that recommendation embeddings are severely spectrally collapsed before attention is applied. The entire forward-collapse bound and the resulting backward gradient-starvation argument depend on this premise, and the paper provides only visual evidence (Figures 2-3) for it. If, after the two-stage warm-up or with different embedding initialization, the spectrum is not dominated in this way, the theoretical motivation for the spectral softening design does not follow, even though the method might still work empirically. Location: Section III, Proposition III.1.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This manuscript addresses the performance gap of standard self-attention in CTR prediction. It argues that heterogeneous and long-tailed recommendation data cause spectral collapse of embeddings, and that this collapse triggers a 'vicious cycle' through forward attention collapse and backward gradient starvation (Section III). It proposes SpecFormer, whose components are learnable spectral softening, spectrum-softened attention, and Taylor-expanded spectral residual position encoding (Section IV). Experiments compare against traditional, Transformer, and spectral baselines on one industrial dataset, Criteo, and Avazu, and report ablations, a depth-scaling study, and an online A/B test. The paper makes both a theoretical claim about why Transformers fail in recommendation and an empirical claim that SpecFormer mitigates collapse and enables depth scaling.

Significance. If proven, the result would be significant for industrial CTR modeling: it identifies a mechanism for the common observation that stacked attention in recommendation underperforms shallower models, and it provides a recipe (spectral softening + spectral biases) that reportedly unlocks depth scaling and large online gains. The paper is also unusually strong in shipping a public code link, giving deployment details, and reporting a real A/B experiment. However, the theoretical propositions are not yet rigorous, the empirical baseline numbers contain implausible entries, and the collapse-mitigation metrics are tied by construction to the proposed architecture. These issues must be resolved before the claims can be accepted.

major comments (5)
  1. [Section III, Proposition III.1] The load-bearing premise σ_k ≤ εσ_1 for all k≥2 is asserted rather than established. Figure 3(b) shows cumulative singular-value energy for OneTrans, but cumulative dominance by the first few components can hold with σ_2/σ_1 = O(1) under a power-law decay; it does not imply that every tail singular value is ε-small. The paper's own Section V-A5 states that randomly initialized embeddings are ill-conditioned, but it never reports the spectrum of the trained embeddings after warm-up. Without this measurement, the forward-collapse theorem does not apply to the setting it is used to justify.
  2. [Section III, Eq. (11)] The bound |R_ij| ≤ O(εσ_1^2) is not justified as stated. The residual sum contains up to r^2−1 terms, each bounded by roughly εσ_1^2 max|M_ab| (or ε^2σ_1^2 when a,b≥2), so the constant depends on r and on M relative to M_11. The proof also does not show the claim that softmax 'exponentially amplifies' the dominance of the rank-1 component; softmax can alter ratios non-trivially and depends on row-wise constants. These gaps are load-bearing because the rest of the 'vicious cycle' relies on A≈A_1.
  3. [Section III, Proposition III.2] The attenuation factor ξ_k = ||A_1^T u_k||/||u_k|| is asserted without proof. The citation to [38] supports rank collapse of attention products, not the specific gradient-attenuation inequality in Eq. (15). Moreover, the expansion in Eq. (14) omits the A_E^T u_1 g_1^T cross term, so the bound in Eq. (15) is not derived; it can only hold under additional assumptions that are not stated. Since the backward-stage argument is half of the 'vicious cycle' claim, this is a central gap.
  4. [Section V-B, Table III] Several baseline numbers are implausible and create doubt about the experimental protocol. DeepFM with AUC 0.5988 on Criteo is far below typical results for that model on this benchmark; FEDIN's Criteo logloss 0.9568 and HiFormer's Avazu logloss 0.4842 are similarly out of line with the other rows. No standard deviations or repeated-run information is reported, although the paper treats differences of 0.001 as significant. The scaling study in Fig. 5 is reported only for the proprietary Industrial dataset, so the main scaling claim is not publicly verifiable. These numbers and the protocol need to be corrected and documented.
  5. [Section IV-A and V-D] The metrics used to demonstrate collapse mitigation—cumulative singular value ratio and fractional effective rank—are exactly the quantities that Learnable Spectral Softening (Eq. 17) and Spectral Residual Position Encoding (Eq. 22) explicitly manipulate by construction. Observing a higher erank for SpecFormer is therefore not independent evidence that the vicious cycle is broken; it may simply reflect that the architecture is engineered to inflate this metric. The paper should provide independent behavioral probes, such as performance on held-out long-tail features, diversity of learned attention patterns, or rank stability under input perturbation, and compare against a control with the same metric-maximizing machinery.
minor comments (4)
  1. [Table II vs. Section V-A1] Table II lists 391 fields for the Industrial dataset, while Section V-A1 says the dataset contains 38 feature fields. Please reconcile this inconsistency.
  2. [Section V-A4] The two-stage warm-up uses a different training protocol from the baselines (5% embedding warm-up + 95% main training vs. single-stage 100% training). Please clarify the total training budget and report sensitivity to the warm-up fraction; otherwise the comparison may be perceived as unfair.
  3. [Figure 2] The heterogeneity claim is based on qualitative visualizations comparing Qwen3 (an LLM) with OneTrans and a single attention heatmap. Quantitative field-separation statistics and aggregated attention statistics would strengthen the motivation.
  4. [Section IV-C, Eqs. (23)-(25)] The Taylor coefficients β_p are described as shared learnable parameters; please state their initialization and clarify whether σ_1^2 in Eq. (25) is treated as a constant or as a gradient-carrying quantity. This affects the interpretation of the ablation in Table IV(e).
Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The architecture's performance depends on fitted/tuned quantities (τ, β, α, γ and the warm-up split) and on the empirical premise that recommendation embedding matrices are spectrally collapsed. The theory also assumes bounded learned projections and a low-pass attention property. No new physical entities are introduced.

free parameters (5)
  • τ (learnable spectral softening exponent) = learned; initialized through a=0 so τ=0.5
    Controls power-law flattening of singular values in Eq. (17); trained end-to-end on CTR loss. Number depends on number of layers.
  • β_0, β_1, β_2 (Taylor expansion coefficients) = not reported (learned)
    Parameterize the spectral residual position encoding in Eq. (24); trained.
  • α (spectral residual weight) = optimal ≈ 0.001
    Hyperparameter controlling P_bias strength; grid-searched over {0.0001, 0.001, 0.01} (Section V-E).
  • γ (spatial residual weight) = optimal ≈ 0.8
    Hyperparameter controlling S_bias strength; grid-searched over {0.5, 0.8, 1.0} (Section V-E).
  • Embedding warm-up fraction = 5%
    Section V-A.5; chosen by hand to stabilize SVD-based softening; no sensitivity analysis.
assumptions (5)
  • domain assumption Input token embedding matrix H satisfies σ_k ≤ εσ_1 for all k≥2 with 0<ε≪1 (severe spectral collapse premise).
    Assumed in Proposition III.1 to represent long-tail recommendation data; the forward-collapse bound and the 'vicious cycle' depend on it. If H is not initially collapsed after warm-up, the theorem's premise fails.
  • domain assumption Learned projection product M = V^T W_q W_k^T V is bounded at a relatively small scale and its dominant entry is not pathologically small.
    Used to argue the term σ1²M11u1u1^T dominates the residual; no bound on M is actually derived from initialization or regularization.
  • standard math Softmax is Lipschitz continuous (cited from [39]).
    Used in Proposition III.2 to bound the difference between A and A1.
  • domain assumption Attention matrices act as low-pass filters on the input spectral basis, attenuating minor spectral components during backpropagation (cited from [38]).
    The backward gradient-starvation argument in Proposition III.2 relies on this property; it is borrowed from prior work rather than proven in this paper.
  • domain assumption Omitting the 1/√D scaling factor in attention does not change the qualitative collapse behavior.
    Stated in Section III before Proposition III.1; a convenience that may affect the quantitative bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation." pith.science (2026). https://pith.science/paper/LXOSLF2H

@misc{pith2026260724025,
  author       = {Pith},
  title        = {Pith review of: SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LXOSLF2H}},
  note         = {Machine review of arXiv:2607.24025}
}
read the original abstract

Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.

Figures

Figures reproduced from arXiv: 2607.24025 by the authors.

Figure 1
Figure 1. The comparison on fractional effective rank of attention [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Visualization of the token semantic space distributions of Qwen3 and OneTrans, indicating evident heterogeneity [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. (a) Token frequency exhibits a severe long-tail dis [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The architecture of SpecFormer. Each SpecFormer attention layer consists of three components: (1) Learnable Spectral [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of AUC gains and attention effective ranks [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Hyperparameter sensitivity analysis of SpecFormer. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 16 linked inside Pith

  1. [38]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth,

    Y . Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” in International conference on machine learning. PMLR, 2021, pp. 2793– 2803

    2021

  2. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  3. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186

    2019

  4. [3]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  5. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  6. [5]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  7. [6]

    Onetrans: Unified feature interaction and sequence modeling with one transformer in industrial recommender,

    Z. Zhang, H. Pei, J. Guo, T. Wang, Y . Feng, H. Sun, S. Liu, and A. Sun, “Onetrans: Unified feature interaction and sequence modeling with one transformer in industrial recommender,” inProceedings of the ACM Web Conference 2026, 2026, pp. 8162–8170

    2026

  8. [7]

    Final: Factorized interaction layer for ctr prediction,

    J. Zhu, Q. Jia, G. Cai, Q. Dai, J. Li, Z. Dong, R. Tang, and R. Zhang, “Final: Factorized interaction layer for ctr prediction,” inProceedings of the 46th International ACM SIGIR conference on research and development in information retrieval, 2023, pp. 2006–2010

    2023

Show all 58 references
  1. [8]

    Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,

    F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang, “Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,” inProceedings of the 28th ACM international confer- ence on information and knowledge management, 2019, pp. 1441–1450

    2019

  2. [9]

    Self-attentive sequential recommenda- tion,

    W.-C. Kang and J. McAuley, “Self-attentive sequential recommenda- tion,” in2018 IEEE international conference on data mining (ICDM). IEEE, 2018, pp. 197–206

    2018

  3. [10]

    Autoint: Automatic feature interaction learning via self-attentive neural networks,

    W. Song, C. Shi, Z. Xiao, Z. Duan, Y . Xu, M. Zhang, and J. Tang, “Autoint: Automatic feature interaction learning via self-attentive neural networks,” inProceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1161–1170

    2019

  4. [11]

    Rankmixer: Scaling up ranking models in industrial recommenders,

    J. Zhu, Z. Fan, X. Zhu, Y . Jiang, H. Wang, X. Han, H. Ding, X. Wang, W. Zhao, Z. Gonget al., “Rankmixer: Scaling up ranking models in industrial recommenders,” inProceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, pp. 6309–6316

    2025

  5. [12]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  6. [13]

    How do recommendation models amplify popularity bias? an analysis from the spectral perspective,

    S. Lin, C. Gao, J. Chen, S. Zhou, B. Hu, Y . Feng, C. Chen, and C. Wang, “How do recommendation models amplify popularity bias? an analysis from the spectral perspective,” inProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, 2025, pp. 659–668

    2025

  7. [14]

    On the embedding collapse when scaling up recommendation models,

    X. Guo, J. Pan, X. Wang, B. Chen, J. Jiang, and M. Long, “On the embedding collapse when scaling up recommendation models,”arXiv preprint arXiv:2310.04400, 2023

    2023 arXiv

  8. [15]

    Rankup: Towards high-rank representa- tions for large scale advertising recommender systems,

    J. Chen, S. Zhang, B. Hu, C. Zhou, J. Pan, G. Xue, W. Ning, G. Weng, W. Zheng, S. Liuet al., “Rankup: Towards high-rank representa- tions for large scale advertising recommender systems,”arXiv preprint arXiv:2604.17878, 2026

    2026 arXiv

  9. [16]

    From scaling to structured expressivity: Rethinking transformers for ctr prediction,

    B. Yan, Y . Lei, Z. Zeng, D. Wang, K. Lin, P. Wang, J. Xu, and B. Zheng, “From scaling to structured expressivity: Rethinking transformers for ctr prediction,”arXiv preprint arXiv:2511.12081, 2025

    2025 arXiv

  10. [17]

    Are id embeddings nec- essary? whitening pre-trained text embeddings for effective sequential recommendation,

    L. Zhang, X. Zhou, Z. Zeng, and Z. Shen, “Are id embeddings nec- essary? whitening pre-trained text embeddings for effective sequential recommendation,” in2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024, pp. 530–543

    2024

  11. [18]

    Fedin: Frequency-enhanced deep interest network for click-through rate pre- diction,

    Z. Dai, J. Wang, J. Pan, D. Liu, L. Xiao, and S.-T. Xia, “Fedin: Frequency-enhanced deep interest network for click-through rate pre- diction,”arXiv preprint arXiv:2605.01726, 2026

    2026 arXiv

  12. [19]

    A convolutional click prediction model,

    Q. Liu, F. Yu, S. Wu, and L. Wang, “A convolutional click prediction model,” inProceedings of the 24th ACM international on conference on information and knowledge management, 2015, pp. 1743–1746

    2015

  13. [20]

    Fibinet: combining feature impor- tance and bilinear feature interaction for click-through rate prediction,

    T. Huang, Z. Zhang, and J. Zhang, “Fibinet: combining feature impor- tance and bilinear feature interaction for click-through rate prediction,” inProceedings of the 13th ACM conference on recommender systems, 2019, pp. 169–177

    2019

  14. [21]

    Field-weighted factorization machines for click-through rate prediction in display advertising,

    J. Pan, J. Xu, A. L. Ruiz, W. Zhao, S. Pan, Y . Sun, and Q. Lu, “Field-weighted factorization machines for click-through rate prediction in display advertising,” inProceedings of the 2018 world wide web conference, 2018, pp. 1349–1357

    2018

  15. [22]

    Fm2: Field-matrixed factor- ization machines for recommender systems,

    Y . Sun, J. Pan, A. Zhang, and A. Flores, “Fm2: Field-matrixed factor- ization machines for recommender systems,” inProceedings of the web conference 2021, 2021, pp. 2828–2837

    2021

  16. [23]

    Deepfm: a factorization- machine based neural network for ctr prediction,

    H. Guo, R. Tang, Y . Ye, Z. Li, and X. He, “Deepfm: a factorization- machine based neural network for ctr prediction,”arXiv preprint arXiv:1703.04247, 2017

    2017 arXiv

  17. [24]

    Fibinet++: Reducing model size by low rank feature interaction layer for ctr prediction,

    P. Zhang, Z. Zheng, and J. Zhang, “Fibinet++: Reducing model size by low rank feature interaction layer for ctr prediction,” inProceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 4425–4429

    2023

  18. [25]

    Factorization machines,

    S. Rendle, “Factorization machines,” in2010 IEEE International con- ference on data mining. IEEE, 2010, pp. 995–1000

    2010

  19. [26]

    Attentional factorization machines: Learning the weight of feature interactions via attention networks,

    J. Xiao, H. Ye, X. He, H. Zhang, F. Wu, and T.-S. Chua, “Attentional factorization machines: Learning the weight of feature interactions via attention networks,”arXiv preprint arXiv:1708.04617, 2017

    2017 arXiv

  20. [27]

    Wide & deep learning for recommender systems,

    H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispiret al., “Wide & deep learning for recommender systems,” inProceedings of the 1st workshop on deep learning for recommender systems, 2016, pp. 7–10

    2016

  21. [28]

    Hiformer: Heterogeneous feature interactions learning with trans- formers for recommender systems,

    H. Gui, R. Wang, K. Yin, L. Jin, M. Kula, T. Xu, L. Hong, and E. H. Chi, “Hiformer: Heterogeneous feature interactions learning with trans- formers for recommender systems,”arXiv preprint arXiv:2311.05884, 2023

    2023 arXiv

  22. [29]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  23. [30]

    Svd approach to data unfolding,

    A. Hoecker and V . Kartvelishvili, “Svd approach to data unfolding,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 372, no. 3, pp. 469–481, 1996

    1996

  24. [31]

    Towards mitigat- ing dimensional collapse of representations in collaborative filtering,

    H. Chen, V . Lai, H. Jin, Z. Jiang, M. Das, and X. Hu, “Towards mitigat- ing dimensional collapse of representations in collaborative filtering,” in Proceedings of the 17th ACM international conference on web search and data mining, 2024, pp. 106–115

    2024

  25. [32]

    Spectran: Spectral-aware transformer-based adapter for llm-enhanced sequential recommendation,

    Y . Cui, F. Liu, Z. Wang, C. Zhang, J. Wang, C. Wang, and J. Chen, “Spectran: Spectral-aware transformer-based adapter for llm-enhanced sequential recommendation,”arXiv preprint arXiv:2601.21986, 2026

    2026 arXiv

  26. [33]

    The effective rank: A measure of effective dimensionality,

    O. Roy and M. Vetterli, “The effective rank: A measure of effective dimensionality,” in2007 15th European signal processing conference. IEEE, 2007, pp. 606–610

    2007

  27. [34]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,”The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948

    1948

  28. [35]

    Signal propagation in transformers: Theoretical perspectives and the role of rank collapse,

    L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Luc- chi, “Signal propagation in transformers: Theoretical perspectives and the role of rank collapse,”Advances in Neural Information Processing Systems, vol. 35, pp. 27 198–27 211, 2022

    2022

  29. [36]

    Diff-erank: A novel rank- based metric for evaluating large language models,

    L. Wei, Z. Tan, C. Li, J. Wang, and W. Huang, “Diff-erank: A novel rank- based metric for evaluating large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 39 501–39 521, 2024

    2024

  30. [37]

    Dcn v2: Improved deep & cross network and practical lessons for web- scale learning to rank systems,

    R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi, “Dcn v2: Improved deep & cross network and practical lessons for web- scale learning to rank systems,” inProceedings of the web conference 2021, 2021, pp. 1785–1797

    2021

  31. [39]

    On the properties of the softmax function with application in game theory and reinforcement learning,

    B. Gao and L. Pavel, “On the properties of the softmax function with application in game theory and reinforcement learning,”arXiv preprint arXiv:1704.00805, 2017

    2017 arXiv

  32. [40]

    Masknet: Introducing feature-wise multiplication to ctr ranking models by instance-guided mask,

    Z. Wang, Q. She, and J. Zhang, “Masknet: Introducing feature-wise multiplication to ctr ranking models by instance-guided mask,”arXiv preprint arXiv:2102.07619, 2021

    2021 arXiv

  33. [41]

    Towards deeper, lighter and interpretable cross network for ctr prediction,

    F. Wang, H. Gu, D. Li, T. Lu, P. Zhang, and N. Gu, “Towards deeper, lighter and interpretable cross network for ctr prediction,” in Proceedings of the 32nd ACM international conference on information and knowledge management, 2023, pp. 2523–2533

    2023

  34. [42]

    Deep interest network for click-through rate prediction,

    G. Zhou, X. Zhu, C. Song, Y . Fan, H. Zhu, X. Ma, Y . Yan, J. Jin, H. Li, and K. Gai, “Deep interest network for click-through rate prediction,” inProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 1059–1068

    2018

  35. [43]

    Adaptive subgradient methods for online learning and stochastic optimization

    J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradient methods for online learning and stochastic optimization.”Journal of machine learning research, vol. 12, no. 7, 2011

    2011

  36. [44]

    A generic learning framework for sequential recommendation with distribution shifts,

    Z. Yang, X. He, J. Zhang, J. Wu, X. Xin, J. Chen, and X. Wang, “A generic learning framework for sequential recommendation with distribution shifts,” inProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 331–340

    2023

  37. [45]

    Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction,

    Q. Pi, G. Zhou, Y . Zhang, Z. Wang, L. Ren, Y . Fan, X. Zhu, and K. Gai, “Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction,” inProceedings of the 29th ACM International Conference on Information & Knowledge Management...

    2020

  38. [46]

    Tallrec: An effective and efficient tuning framework to align large language model with recommendation,

    K. Bao, J. Zhang, Y . Zhang, W. Wang, F. Feng, and X. He, “Tallrec: An effective and efficient tuning framework to align large language model with recommendation,” inProceedings of the 17th ACM Conference on Recommender Systems, 2023, pp. 1007–1014

    2023

  39. [47]

    Hllm: Enhancing sequential recommendations via hierarchical large language models for item and user modeling,

    J. Chen, L. Chi, B. Peng, and Z. Yuan, “Hllm: Enhancing sequential recommendations via hierarchical large language models for item and user modeling,”arXiv preprint arXiv:2409.12740, 2024

    2024 arXiv

  40. [48]

    Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation,

    J. Lin, R. Shan, C. Zhu, K. Du, B. Chen, S. Quan, R. Tang, Y . Yu, and W. Zhang, “Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation,” inProceedings of the ACM on Web Conference 2024, 2024, pp. 3497–3508

    2024

  41. [49]

    Breaking the length barrier: Llm-enhanced ctr prediction in long textual user behaviors,

    B. Geng, Z. Huan, X. Zhang, Y . He, L. Zhang, F. Yuan, J. Zhou, and L. Mo, “Breaking the length barrier: Llm-enhanced ctr prediction in long textual user behaviors,” inProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retriev...

    2024

  42. [50]

    Ctrl: Connect collaborative and language model for ctr prediction,

    X. Li, B. Chen, L. Hou, and R. Tang, “Ctrl: Connect collaborative and language model for ctr prediction,”ACM Transactions on Recommender Systems, vol. 4, no. 2, pp. 1–23, 2025

    2025

  43. [51]

    Towards open-world recommendation with knowledge augmentation from large language models,

    Y . Xi, W. Liu, J. Lin, X. Cai, H. Zhu, J. Zhu, B. Chen, R. Tang, W. Zhang, and Y . Yu, “Towards open-world recommendation with knowledge augmentation from large language models,” inProceedings of the 18th ACM Conference on Recommender Systems, 2024, pp. 12– 22

    2024

  44. [52]

    Large language models enhanced collaborative filtering,

    Z. Sun, Z. Si, X. Zang, K. Zheng, Y . Song, X. Zhang, and J. Xu, “Large language models enhanced collaborative filtering,” inProceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 2178–2188

    2024

  45. [53]

    Field matters: A lightweight llm-enhanced method for ctr prediction,

    Y . Cui, F. Liu, J. Chen, X. Lou, C. Zhang, J. Wang, Y . Sun, X. Yang, and C. Wang, “Field matters: A lightweight llm-enhanced method for ctr prediction,” inProceedings of the ACM Web Conference 2026, 2026, pp. 6365–6376

    2026

  46. [54]

    Representation degeneration problem in training natural language generation models,

    J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y . Liu, “Representation degeneration problem in training natural language generation models,” arXiv preprint arXiv:1907.12009, 2019

    1907 arXiv

  47. [55]

    Understanding dimen- sional collapse in contrastive self-supervised learning,

    L. Jing, P. Vincent, Y . LeCun, and Y . Tian, “Understanding dimen- sional collapse in contrastive self-supervised learning,”arXiv preprint arXiv:2110.09348, 2021

    2021 arXiv

  48. [56]

    On feature decorrelation in self-supervised learning,

    T. Hua, W. Wang, Z. Xue, S. Ren, Y . Wang, and H. Zhao, “On feature decorrelation in self-supervised learning,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9598– 9608

    2021

  49. [57]

    Alphafuse: Learn id embeddings for sequential recommendation in null space of language embeddings,

    G. Hu, A. Zhang, S. Liu, Z. Cai, X. Yang, and X. Wang, “Alphafuse: Learn id embeddings for sequential recommendation in null space of language embeddings,” inProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025, ...

    2025

  50. [58]

    Low-rank bottleneck in multi-head attention models,

    S. Bhojanapalli, C. Yun, A. S. Rawat, S. Reddi, and S. Kumar, “Low-rank bottleneck in multi-head attention models,” inInternational conference on machine learning. PMLR, 2020, pp. 864–873

    2020

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.