Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Scaling Transformers for Discriminative Recommendation via Generative Pretraining

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A generative-pretraining step that freezes sparse embeddings turns overfitting ranking Transformers into predictably scalable ones.

desk verdict A practical, industry-grounded paper whose sparse-freezing trick clearly works; the scaling-law claim is real but under-verified because it rests on single-epoch training. read the letter →

arxiv 2506.03699 v2 pith:FWAKWJME submitted 2025-06-04 cs.IR

Chunqi Wang , Bingchao Wu , Zheng Chen , Lei Shen , Bing Wang , Xiaoyi Zeng This is my paper
classification cs.IR
keywords recommendersystemsgenerativepretrainingdiscriminativerecommendationCTRpredictionscalinglawTransformersparseparameterfreezingoverfitting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative pretraining combined with a frozen embedding table solves an overfitting problem that has kept Transformer ranking models small, the paper argues. Discriminative CTR/CVR models trained from scratch show two distinct overfitting modes, one at epoch boundaries and one within the first epoch, so larger models can validate worse than smaller ones. GPSD first pretrains a Transformer to predict the next item in user behavior sequences, then transfers the learned sparse embedding parameters into a discriminative ranking model and freezes them while training only dense parameters. With this recipe, the generalization gap narrows, and as dense parameters grow from 13K to 0.3B, AUC improves steadily and fits a power law with an empirical ceiling near 0.7097 on a 5-billion-sample industrial dataset. If the claim holds, ranking Transformers can be scaled predictably in the same way language models are.

What carries the argument

The machinery is the GPSD framework with its Sparse Transfer & Sparse Freeze bridging strategy: a unidirectional Transformer is pretrained autoregressively to predict the next item in user behavior sequences using sampled softmax over a vocabulary of millions of item IDs, with input and output embeddings tied. The learned embedding table is then loaded into a discriminative ranking Transformer and frozen, while only the dense Transformer and MLP parameters are updated with cross-entropy loss. This separation of sparse and dense parameters is what lets generative pretraining do its work: the frozen embeddings carry stable user-item representations, and the trainable dense parameters are free to scale without the one-epoch and within-one-epoch overfitting that otherwise appear.

What would settle it

Train the smaller models in the scaling sweep, such as L1H32A4 and L4H32A4, on CTR-XL for two or three epochs under the same ST&SF strategy; if their AUC rises to meet or exceed the largest model's 0.7018, the fitted power law is an artifact of uniform under-training rather than a genuine capacity scaling law.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in scaling discriminative recommendation Transformers is not the Transformer itself but the sparse embedding parameters, which overfit when trained discriminatively on sparse user behavior data. The paper shows that generative pretraining with heavy random negative sampling trains these embeddings stably, and that transferring them into a discriminative model and freezing them removes the overfitting that blocks scaling. Under this Sparse Transfer & Sparse Freeze strategy, the dense Transformer parameters scale monotonically from 13K to 327M parameters on CTR-XL, with AUC rising from 0.6306 to 0.7018 and loss falling from 0.3922 to 0.3732, both closely following fitted power laws. Merely transferring all parameters without freezing, as in standard language-model fine-tuning, does not fix the problem, which the paper takes as evidence that sparse-parameter learning is the core obstacle.

Load-bearing premise

The scaling law is estimated from models trained for exactly one epoch on CTR-XL, because Section 3.6 says one epoch was used due to resource constraints, so the claim that larger models are genuinely better assumes one epoch is enough for every model size to reveal its true capacity.

Editorial extensions

If this is right

  • Transformer-based ranking models can be scaled predictably like language models, with AUC and loss following fitted power laws in dense parameter count, instead of plateauing or degrading at larger sizes.
  • Pretrained frozen sparse embeddings behave as portable assets: the paper shows they improve scalability even when transferred to non-Transformer architectures such as HSTU and Wukong.
  • Even a small ranking Transformer, 3 layers wide by 160 hidden units, delivers large online gains under incremental training, with reported increases of 7.03% in GMV, 2.11% in orders, 1.86% in buyers, and 3.78% in CTR.
  • The Sparse Transfer & Sparse Freeze recipe also improves existing non-Transformer ranking models on public datasets, with relative AUC gains between 2.36% and 17.31%, so the benefit is not limited to Transformer-only pipelines.
  • The fitted power law gives practitioners a way to forecast the payoff of a larger dense model before spending GPU resources, such as the empirical AUC ceiling near 0.7097 on CTR-XL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because every point on the scaling curve was trained for exactly one epoch, the power law may reflect per-epoch learning behavior rather than true capacity; testing small models with two or three epochs would show whether the fitted curve is a real capacity law or an artifact of uniform under-training.
  • The empirical AUC ceiling near 0.7097 comes from scaling dense parameters alone, so further gains on large ranking datasets may require unfreezing a subset of sparse parameters, adding features, lengthening sequences, or improving data quality, directions the paper itself leaves open.
  • The success of cross-architecture transfer suggests the generatively pretrained embedding table encodes reusable user-interest structure independent of the dense architecture, which could be tested by attaching the same frozen embeddings to a variety of dense heads and measuring how much performance transfers.
  • If the sparse-freeze principle generalizes, it may change how recommendation models are pretrained: instead of matching architectures exactly, practitioners could invest in high-quality generative sparse parameters once and share them across many ranking models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes GPSD, a two-stage framework for discriminative recommendation (CTR/CVR/CART). A Transformer is first trained generatively with sampled softmax on user behavior sequences; the learned parameters are then transferred to a discriminative ranking model, with the sparse embedding parameters frozen (the ST&SF strategy being preferred). Experiments cover three industrial tasks, an expanded 5B-sample CTR-XL dataset, three public datasets, cross-architecture transfer to HSTU and Wukong, and a 30-day online A/B test. The headline claim is that, with this initialization and freezing scheme, Transformer ranking models scale consistently in dense parameters from 13K to 0.3B following a power law (AUC asymptote 0.7097, loss asymptote 0.3695), in contrast to from-scratch discriminative training, which overfits and does not scale.

Significance. If the scaling-law claim holds, the paper would be an important step toward making Transformer ranking models as scalable as language models, and the sparse-freeze transfer recipe is simple and likely to be adopted. The paper's strengths are the breadth of evidence: three industrial datasets plus a 5B-sample dataset, public benchmarks, cross-architecture transfer, an online A/B test, and released code. The observation that freezing transferred embeddings removes both one-epoch and within-one-epoch overfitting is interesting in its own right. However, the scaling-law evidence is currently descriptive rather than validated predictive scaling, and a confound between dense and sparse parameter growth weakens the causal reading of the central claim.

major comments (3)
  1. [§3.6, Figure 5, Table 8; also §3.2] The scaling law is derived entirely from models trained for exactly one epoch on CTR-XL, as stated in §3.6 ('we train each model for one epoch due to resource constrain') and in Table 2. A scaling law that is claimed to characterize the model family should be approximately invariant to the training budget, at least in the ordering of models and in the located asymptote. The paper provides no multi-epoch results on CTR-XL for any model size. Because §3.3 and Figure 4 show that validation AUC is still changing at epoch boundaries and that generative training does not overfit, one epoch is not established as a canonical or sufficient operating point across all scales. If smaller models catch up with additional epochs, the fitted power law and the derived bounds (AUC ≈ 0.7097, loss ≈ 0.3695) would be artifacts of a fixed training schedule rather than evidence of a genuine scaling law. Please report multi-epoch runs for at least the smaller model sizes, or otherwise show that the one-epoch curves already place the models at the same relative positions as at convergence. In addition, the power law is fitted to all eight points in Figure 5; a stronger claim requires fitting on the first k sizes and predicting the remaining ones, as is standard in scaling-law validation.
  2. [§3.6, Table 5] The scaling experiment varies dense and sparse parameters simultaneously. Table 5 shows #Sparse Params growing from 125M to 4B while #Dense Params grows from 13K to 327M, with both tied to model width. Because the method's key ingredient is the frozen sparse embedding table, larger models also receive larger and higher-capacity embeddings. The central claim that performance scales with 'dense parameters' is therefore not identified as an effect of dense parameters alone. Please hold the sparse parameter count fixed while varying dense parameters (or decouple embedding width from Transformer width), or at least present an analysis that separates the contribution of embedding width from the contribution of Transformer depth and width.
  3. [Table 6, Table 7, Figure 5] No uncertainty quantification is provided for any of the central empirical results. Public dataset results in Table 6 are single AUC values without standard deviations over seeds; the online A/B gains in Table 7 are point estimates without confidence intervals or significance tests; and the scaling-curve points in Figure 5 are single runs. Given that some reported improvements are modest (e.g., DeepFM + ST&SF on Taobao, +5.56%), this makes it difficult to assess whether the observed orderings and gains are stable. Please add repeated-seed results for the public datasets and at least confidence intervals or significance tests for the online metrics.
minor comments (4)
  1. [Figure 5] The power-law equations are rendered with '□' placeholders instead of minus signs (e.g., 'y = 0.7097 □ 0.7553x□0.2364'); please fix the typography.
  2. [§3.8, §4.3, §3.9, §3.3] There are several typos: 'perfromance' (§3.8), 'Addtionally' (§4.3), 'new comming' (§3.9), and 'phenomenons' (§3.3).
  3. [Table 3 caption] The caption says 'The underline denote the best performance' and 'bold denote the best'; these should be 'The underline denotes' and 'bold denotes'.
  4. [§3.2] The notation LuHvAw is used but never formally defined in the text; please define it explicitly, even though the meaning is inferable from Table 5.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: GPSD's core transfer and sparse-freezing results are evaluated on held-out labels and external benchmarks, and the scaling law is honestly presented as an estimated power-law fit rather than a prior prediction.

full rationale

The central GPSD claim is not circular. Generative pretraining produces a set of parameters; discriminative models are then evaluated on held-out validation/test splits and online A/B tests, with the five transfer strategies compared against a from-scratch baseline. The scaling-law statement in Section 3.6 is explicitly an empirical fit: 'the power laws can fit the observations as shown by dashed lines in the figure' and 'the estimated power laws also tell us the empirical upper bound of AUC.' This is curve fitting, not a prediction forced by construction; the fitted power law and its asymptotic bounds are parameters of the fit, not independent claims derived from the same inputs. The one-epoch training budget and per-scale learning-rate changes on CTR-XL are legitimate validity concerns about whether the scaling curve reflects a true model-family law, but they are not circularity under the definitions used here. The only self-citation found, [7] (ZEUS), is used to support the incidental observation that earlier Transformer ranking models used single layers, and it is not load-bearing for the paper's main argument. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is merely renamed. Therefore the derivation chain is self-contained against external benchmarks and the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard sampled-softmax assumptions, a transfer assumption between clicked-item sequences and impression-based CTR labels, and a chosen functional form for the scaling law. The fitted power-law constants are descriptive, not derived. No new physical or conceptual entities are introduced.

free parameters (3)
  • AUC power-law asymptote a = 0.7097
    Fitted in Figure 5a as the empirical upper bound of AUC on CTR-XL; the central 'scaling law' claim depends on this fit.
  • AUC power-law coefficient b and exponent c = 0.7553, 0.2364
    Fitted to the eight scaling observations in Figure 5a; the power-law shape is assumed, not derived.
  • Loss power-law asymptote a and coefficients b, c = 0.3695, 0.1420, 0.1915
    Fitted in Figure 5b to the validation loss scaling curve; descriptive fit rather than a theoretical prediction.
assumptions (4)
  • standard math Sampled softmax with uniform negative sampling approximates full softmax and permits omission of the correction term.
    Used in Section 2.1 to define the generative pretraining loss; the omitted correction term is constant under uniform sampling, which is valid but unproved in the paper.
  • domain assumption The generative pretraining corpus CLICK, built from clicked-item sequences, transfers to the discriminative tasks CTR/CVR/CART despite different label semantics.
    Section 3.1 constructs CLICK from clicked items; the claim that these embeddings help discriminative ranking depends on this transfer assumption.
  • domain assumption AUC with one random negative per positive is a reliable evaluation metric for ranking.
    Used throughout Section 3; standard in the field but a simplification of real-world ranking error.
  • ad hoc to paper Model performance follows the functional form y = a - b * x^(-c) in dense parameter count.
    In Section 3.6 and Figure 5, the scaling law is assumed to take this three-parameter form; no derivation from theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Transformers for Discriminative Recommendation via Generative Pretraining." pith.science (2026). https://pith.science/paper/FWAKWJME

@misc{pith2026250603699,
  author       = {Pith},
  title        = {Pith review of: Scaling Transformers for Discriminative Recommendation via Generative Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWAKWJME}},
  note         = {Machine review of arXiv:2506.03699}
}
read the original abstract

Discriminative recommendation tasks, such as CTR (click-through rate) and CVR (conversion rate) prediction, play critical roles in the ranking stage of large-scale industrial recommender systems. However, training a discriminative model encounters a significant overfitting issue induced by data sparsity. Moreover, this overfitting issue worsens with larger models, causing them to underperform smaller ones. To address the overfitting issue and enhance model scalability, we propose a framework named GPSD (\textbf{G}enerative \textbf{P}retraining for \textbf{S}calable \textbf{D}iscriminative Recommendation), drawing inspiration from generative training, which exhibits no evident signs of overfitting. GPSD leverages the parameters learned from a pretrained generative model to initialize a discriminative model, and subsequently applies a sparse parameter freezing strategy. Extensive experiments conducted on both industrial-scale and publicly available datasets demonstrate the superior performance of GPSD. Moreover, it delivers remarkable improvements in online A/B tests. GPSD offers two primary advantages: 1) it substantially narrows the generalization gap in model training, resulting in better test performance; and 2) it leverages the scalability of Transformers, delivering consistent performance gains as models are scaled up. Specifically, we observe consistent performance improvements as the model dense parameters scale from 13K to 0.3B, closely adhering to power laws. These findings pave the way for unifying the architectures of recommendation models and language models, enabling the direct application of techniques well-established in large language models to recommendation models. The code is available at https://github.com/chqiwang/gpsd-rec.

Figures

Figures reproduced from arXiv: 2506.03699 by the authors.

Figure 1
Figure 1. Illustration of the overfitting phenomenon (left) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the GPSD framework. The left part shows the generative and discriminative models based on the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A comparison of training behaviors between gener [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: AUC curves of discriminative models for the CTR [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results of scaling. Solid circles show the factual performance while the green dashed lines denotes the estimated [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Illustration of the incremental-GPSD framework. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Models

    cs.IR 2025-07 unverdicted novelty 3.0 of 10

    A survey of LLM-based generative recommendation systems, covering application settings, training pipelines, industrial deployment challenges, and future directions.

Reference graph

Works this paper leans on

44 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. 2022. Understanding scaling laws for recommendation models. arXiv preprint arXiv:2208.08489 (2022)

    arXiv 2022

  2. [2]

    Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. InProceedings of the 1st international workshop on deep learning practice for high-dimensional sparse data. 1–4

    2019

  3. [3]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy...

    2019

  4. [4]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    arXiv 2020

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  6. [6]

    Zhongxiang Fan, Zhaocheng Liu, Jian Liang, Dongying Kong, Han Li, Peng Jiang, Shuang Li, and Kun Gai. 2024. Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction. arXiv preprint arXiv:2407.01607 (2024)

    arXiv 2024

  7. [7]

    Yulong Gu, Wentian Bao, Dan Ou, Xiang Li, Baoliang Cui, Biyu Ma, Haikuan Huang, Qingwen Liu, and Xiaoyi Zeng. 2021. Self-supervised learning on users’ spontaneous behaviors for multi-scenario ranking in e-commerce. In Proceedings of the 30th ACM International Conference on Information & Knowledge Manage- ment. 3828–3837

    work page 2021

  8. [8]

    Yulong Gu, Zhuoye Ding, Shuaiqiang Wang, Lixin Zou, Yiding Liu, and Dawei Yin

Show all 44 references
  1. [9]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction.arXiv preprint arXiv:1703.04247 (2017)

    2017 arXiv

  2. [10]

    Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jie Jiang, and Mingsheng Long. 2023. On the Embedding Collapse when Scaling up Recommendation Models. arXiv preprint arXiv:2310.04400 (2023)

    2023 arXiv

  3. [11]

    Ruidong Han, Qianzhong Li, He Jiang, Rui Li, Yurou Zhao, Xiang Li, and Wei Lin. 2024. Enhancing CTR Prediction through Sequential Recommendation Pre- training: Introducing the SRP4CTR Framework. In Proceedings of the 33rd ACM International Conference on Information and Knowled...

    2024

  4. [12]

    B Hidasi. 2015. Session-based Recommendations with Recurrent Neural Networks. arXiv preprint arXiv:1511.06939 (2015)

    2015 arXiv

  5. [13]

    Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007 (2014)

    2014 arXiv

  6. [14]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In 2018 IEEE international conference on data mining (ICDM) . IEEE, 197–206

    2018

  7. [15]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  8. [16]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  9. [17]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101 [cs.LG] https://arxiv.org/abs/1711.05101

    2019 arXiv

  10. [18]

    Ze Lyu, Yu Dong, Chengfu Huo, and Weijun Ren. 2020. Deep match to rank model for personalized click-through rate prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 156–163

    2020

  11. [19]

    Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, et al. 2022. Software-hardware co-design for fast and scalable training of deep learning rec- ommendation models. In Proceedings of th...

    2022

  12. [20]

    Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Practice on long sequential user behavior modeling for click-through rate prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 2671–2679

    2019

  13. [21]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Brad- bury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems 5 (2023), 606–624

    2023

  14. [22]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Im- proving language understanding by generative pre-training. (2018)

    2018

  15. [23]

    Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 (2020)

    2020 arXiv

  16. [24]

    Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlén Ramström, Jisu Jeong, Jung-Woo Ha, and Kyung-Min Kim. 2023. Scaling law for recommendation models: Towards general-purpose user representations. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 4596–4604

    2023

  17. [25]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063

    2024

  18. [26]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang

  19. [27]

    Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommenda- tion via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining . 565–573

    2018

  20. [28]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  21. [29]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, ...

    2017

  22. [30]

    Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, and Tianyu Qiu. 2024. On the effectiveness of sampled softmax loss for item recommendation. ACM Transactions on Information Systems 42, 4 (2024), 1–26

    2024

  23. [31]

    Zhibo Xiao, Luwei Yang, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. 2020. Deep multi-interest network for click-through rate prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management . 2265–2268

    2020

  24. [32]

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In International Conference on Machine Learning. PMLR, 10524–10533

    2020

  25. [33]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  26. [34]

    Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020. Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retri...

    2020

  27. [35]

    Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhao- jie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152 (2024)

    2024 arXiv

  28. [36]

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12104–12113

    2022

  29. [37]

    Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. 2024. Wukong: Towards a Scaling Law for Large-Scale Recommendation. arXiv preprint arXiv:2403.02545 (2024)

    2024 arXiv

  30. [38]

    Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems 32 (2019)

    2019

  31. [39]

    Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji- Rong Wen. 2024. Scaling law of large sequential recommendation models. In Proceedings of the 18th ACM Conference on Recommender Systems . 444–453

    2024

  32. [40]

    Zhao-Yu Zhang, Xiang-Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng, and Bo Zheng. 2022. Towards understanding the overfitting phenomenon of deep click-through rate models. In Proceedings of the 31st ACM international conference on information & knowledge mana...

    2022

  33. [41]

    Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 5941–5948

    2019

  34. [42]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...

    2018

  35. [2019]

    In Proceedings of the 28th ACM international conference on information and knowledge management

    BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management . 1441–1450

  36. [2020]

    In Proceedings of the 29th ACM international conference on information & knowledge management

    Deep multifaceted transformers for multi-objective ranking in large-scale e-commerce recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management . 2493–2500

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.