Pith. sign in

REVIEW 3 major objections 5 minor 41 references

WHALE, a layer-wise fusion of Wukong-style feature interaction and HSTU-style sequence modeling, consistently outperforms models that scale either paradigm alone, and the advantage grows as compute increases.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:12 UTC pith:WTYZ2CHJ

load-bearing objection Solid, deployable industrial paper: per-layer Wukong-HSTU fusion is a real engineering contribution with online gains, but the FLOPs-alignment claim is oversold unless the K/V amortization is applied symmetrically to baselines. the 3 major comments →

arxiv 2607.17017 v2 pith:WTYZ2CHJ submitted 2026-07-19 cs.IR cs.AIcs.LG

WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture

Renqin Cai , Dawei Sun , Yuanjun Yao , Zhiyong Wang , Velvin Fu , Maggie Zhuang , Yu Shi , Zhongnan Fang
show 3 more authors
Xuan Cao Jing Qian Rui Li
This is my paper
classification cs.IR cs.AIcs.LG
keywords recommender systemsfeature interactionsequence modelingWukongHSTUunified architecturescalabilityindustrial deployment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that the next stage of scaling industrial recommenders can come from unifying two complementary architectures rather than enlarging either one. WHALE stacks layers in which a Wukong module (a factorization-machine backbone for high-order feature crosses) models user, item, and context features, an HSTU module (a sequence-transduction backbone) models the ordered behavior history, and an attention module lets each feature cross query the history and retrieve the behavior evidence most relevant to the current candidate and context. Repeating this exchange at every layer keeps both branches active and lets feature combinations refine against the history progressively. On a large industrial short-form video dataset, WHALE reports better normalized entropy (NE, lower is better) than models that scale either branch alone at matched compute, with its advantage widening as compute grows, plus monotonic gains from longer histories, more layers, and wider embeddings; online A/B tests show positive gains with a 5% throughput regression. The authors read these results as evidence that progressive cross-branch exchange is a practical, deployable scaling principle.

Core claim

WHALE's discovery is that keeping a feature-interaction branch and a sequence branch simultaneously active, and fusing them with attention at every layer, yields a scalable recommendation architecture whose quality improves with model size in ways neither branch achieves alone. Each WHALE layer computes a Wukong representation of non-sequence user, item, and context interactions, an HSTU representation of the ordered behavior history, and a fused representation obtained by letting every Wukong-derived vector query the HSTU-derived history, with the Wukong side providing queries and the HSTU side providing keys and values, followed by a fusion MLP and a SwiGLU feed-forward network. The fused

What carries the argument

The load-bearing object is the WHALE layer: a recursive unit containing a Wukong module (a stacked factorization-machine block plus linear compression that models high-order non-sequence feature crosses), an HSTU module (a sequential attention block with relative position-and-time biases that models long behavior histories), and an attention-based fusion module. In the fusion module, the Wukong output is normalized and projected into queries; the HSTU output is normalized and projected into shared keys and values; a softmax attention produces one history-derived vector per interaction representation; a fusion MLP concatenates that vector with the Wukong output and adds a residual; and a pre-

Load-bearing premise

Everything hinges on whether the FLOPs used to compare the three models count each architecture's work fairly. The paper's own footnote says the reported FLOPs include request-only computation and that WHALE computes its K/V projections once per request and amortizes them across many candidates; if that accounting favors WHALE relative to the baselines, the claim of winning at equal complexity is not settled, though the raw quality and online gains could still be real.

What would settle it

Recompute the main quality-complexity comparison with a stricter cost model: charge every candidate its full share of the history-side K/V computation (no request-level amortization) and recompute FLOPs for all three architectures under identical accounting, or instead match the models on parameter count and layer count with amortization disabled. If the WHALE advantage disappears or reverses under that accounting, the central complexity-alignment claim is refuted; if it persists, the claim survives. A second, complementary check would be an offline experiment that evaluates WHALE on requests

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If WHALE is right, the scaling playbook for large recommenders shifts: adding a second, complementary paradigm and exchanging across layers buys more quality at the same FLOPs than enlarging either paradigm alone, and the benefit grows with compute.
  • Longer user histories, more layers, and wider embeddings all keep paying off across the tested ranges (15k history steps, 8 layers, width 512), suggesting WHALE can absorb continued capacity increases without saturating at the evaluated scales.
  • The ablation results imply that the attention mechanism's selective retrieval is doing the work: replacing it with average pooling over the history costs 0.23% NE, and compressing history into a shallow summary before feature interaction costs 0.25% NE, so fine-grained progressive fusion is worth a substantial fraction of the reported gains.
  • The deployment results imply the architectural benefit can survive serving constraints: online, the primary metric rises 0.113% and two supporting metrics rise 0.824% and 1.820%, at the price of a 5% inference QPS regression that the authors judged acceptable under the serving budget.
  • Because the advantage over single-paradigm baselines widens as FLOPs grow, the unified design is a scaling property rather than a one-time quality bump; this predicts WHALE will continue to pull ahead as models are scaled further.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: check whether the fusion-attention gains concentrate on candidates with sparse static features or long-tail items, where selective retrieval from history should matter most; if gains are uniform, an alternative explanation such as raw capacity increase would be more plausible.
  • WHALE's efficiency story depends on request-level reuse of the history computation across many candidates, which is characteristic of short-form video ranking. The same layer-wise fusion would be much more expensive on surfaces with one or few candidates per request, so the design's deployability may not transfer directly to those settings.
  • The shared key/value and shared-gate choices are reported as quality-neutral in this setting; a reader could probe whether the tie-breaking remains neutral with wider embeddings or with histories containing heterogeneous action types, since tying removes degrees of freedom that could matter at higher capacity.
  • By framing the open question as 'which fusion granularity pays off,' the architecture suggests a spectrum from one-shot summaries to per-layer attention; a natural next experiment is a controlled study of fusion frequency (e.g., every other layer) to find the minimum exchange required for the observed gains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes WHALE, a stacked recommendation architecture that keeps a Wukong-style feature-interaction branch and an HSTU-style sequence branch active at every layer. Each WHALE layer contains a Wukong module, an HSTU module, and an attention-based fusion module in which non-sequence interaction representations query fine-grained behavior-history representations; the fused output is refined by a residual MLP projection and a SwiGLU FFN before being passed to the next layer. The authors also describe training and inference optimizations: a fused Triton attention kernel with shared key/value, Q-parallel and KV-parallel backward variants, a shared-gate SwiGLU FFN, mixed-precision execution, AOTInductor, and shape-hint tensors to avoid CPU-GPU synchronization. On large proprietary industrial data, WHALE reports consistent NE gains over Wukong-only and HSTU-only baselines at matched FLOPs, monotonic improvements when scaling sequence length, depth, and width, ablation results favoring attention-based progressive fusion over shallow-hybrid and average-pooling baselines, and a positive online A/B result with a 5% inference QPS regression. The central claim is that keeping both backbones active and exchanging information at every layer is a practical and scalable unification principle.

Significance. If the FLOPs-based complexity alignment is fair, the paper makes a meaningful practical contribution: it provides a concrete industrial architecture that unifies two complementary scaling paradigms and shows via ablations that the layer-wise attention-based exchange, rather than simply adding capacity, is responsible for the gains. The paper is honest about the serving trade-off and includes detailed system optimizations, and the deployment evidence is valuable. Its main strength is the coherence of the design: the equations are clear, the ablations isolate the proposed components, and the online A/B result shows real-world impact. The principal weakness is that the headline comparison against single-paradigm baselines depends on an underspecified FLOPs amortization rule; until that accounting is clarified, the claim of outperforming at comparable complexity is not fully established.

major comments (3)
  1. [§5.1, §5.2, Footnote 1] The headline comparison in Fig. 3 ('outperforms at comparable complexity') relies on FLOPs-based complexity alignment. Footnote 1 states that FLOPs include request-only computations and that WHALE's K/V projections are amortized via M-FALCON across 'hundreds of examples,' but it never states the candidate count C, nor whether the same amortization is applied to the Wukong-only and HSTU-only baselines. Wukong-only has no request-shared sequence branch, so if its per-example FLOPs are unamortized while WHALE's are divided by C, the matched-FLOPs curves do not represent equal per-request serving cost. The 5% inference QPS regression in Table 2 is consistent with WHALE being more expensive per request. Please report per-request FLOPs, C, QPS/latency at matched FLOPs, and use identical amortization rules across all architectures.
  2. [§5.5, Table 2] The online A/B test compares WHALE against an unspecified 'online baseline.' It is not stated whether this baseline is Wukong-only, HSTU-only, or a different production model. Without this information, the online lift of +0.113% cannot be attributed to the unified Wukong–HSTU design; it may simply reflect an improved production system. Please identify the baseline architecture and, if available, include A/B comparisons against the single-paradigm systems used offline.
  3. [§5.1, §5.2, Table 1] All offline results are single runs without confidence intervals or significance tests. The paper states that a 0.05% NE gain is considered noticeable; several ablation differences in Table 1 are 0.08% and 0.11%, and the Fig. 3 curves have no error bars. At these effect sizes, run-to-run variance could change the interpretation of which fusion components matter. Please provide repeated runs, standard deviations, or significance tests for at least the headline comparisons and the main ablation rows.
minor comments (5)
  1. [Eq. (5)] The symbol A is reused for both the attention output in Eq. (4) and the fused output after the residual MLP projection in Eq. (5). Using a distinct symbol such as F would improve readability.
  2. [§5.1, Footnote 1] The phrase 'FLOPs reported in this paper include request-only computations' is ambiguous with the earlier statement that FLOPs denote per-example forward-pass computation. Clarify whether per-example FLOPs are computed as (request-only FLOPs / number of candidates) plus per-candidate FLOPs, and state the candidate count used.
  3. [Eq. (5)] The 'Fusion MLP' is a single linear projection of concatenated vectors, not a multi-layer MLP. Consider renaming it to 'fusion projection' or describing it as a one-layer MLP.
  4. [§5.4, Table 1] The ablation baseline is WHALE itself, and all NE regressions are relative to it. Reporting the absolute NE of the reference would help readers gauge the scale of the effects.
  5. [§5.3, Fig. 4] The scaling experiments vary only one capacity axis at a time and report no error bars. It would strengthen the scaling claim to include at least one comparison against a single-paradigm baseline at the same FLOPs, especially at the largest width.

Circularity Check

0 steps flagged

No circular derivation: results are empirical; FLOPs amortization is a fairness caveat, not a construction.

full rationale

WHALE is an empirical architecture paper. Its central claims—consistent offline gains over Wukong-only and HSTU-only baselines at comparable complexity, scaling with sequence length/depth/width, and positive online A/B results—are measured outcomes, not quantities derived from the model equations. Equations (1)–(7) define the architecture; they do not contain the experimental results, so there is no fitted parameter renamed as a prediction and no self-definitional reduction. The main circular-adjacent concern is Footnote 1's FLOPs accounting: WHALE's K/V projections are amortized per request via M-FALCON [34], and the paper does not state whether the Wukong-only and HSTU-only baselines receive the same request-level amortization. If they do not, the 'comparable complexity' comparison undercounts WHALE's per-request serving cost, and the reported 5% inference QPS regression is consistent with that possibility. However, this is a measurement/fairness caveat rather than circularity: the NE gains and A/B lifts are observed outcomes independent of the FLOPs count, and the comparison does not define WHALE's quality in terms of the FLOPs alignment. The cited HSTU and Wukong works share some authors with this paper, but they are published external architectures and the present results are tested against implemented baselines plus a production A/B test, so the self-citations are not load-bearing in a way that forces the conclusions.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are posited; WHALE is a software architecture. There are no fitted constants in a derivation sense; the reported numbers are empirical measurements, and the only hand-chosen items are architectural hyperparameters (depth, width, sequence length) that are explicitly varied in scaling studies.

axioms (5)
  • domain assumption Wukong and HSTU block equations (Eq. 1 and Eq. 2) correctly describe the two backbones and their scaling behavior.
    WHALE is built on these equations; if the backbones are mis-specified, the unified model is not what it claims.
  • ad hoc to paper Per-request M-FALCON amortization of K/V projections is a valid way to report FLOPs and to compare complexity across models.
    Footnote 1; used to align Wukong-only, HSTU-only, and WHALE FLOPs in Fig.3.
  • domain assumption User behavior can be represented as ordered sequences of item+side-information embeddings of length L up to 15k.
    Section 3.2; if the sequence encoding loses signal, the HSTU branch cannot deliver evidence.
  • standard math Standard transformer math (softmax attention, layernorm, residual, SwiGLU) is taken as background.
    Eq. 4-6 rely on standard attention and normalization; no new theory.
  • domain assumption The NE metric with platform significance thresholds is a valid proxy for recommendation quality.
    Sec 5.1; central evaluation relies on 0.05%/0.03% thresholds.

pith-pipeline@v1.3.0-alltime-deepseek · 14961 in / 15896 out tokens · 165243 ms · 2026-08-04T04:12:11.428797+00:00 · methodology

0 comments
read the original abstract

As scalability becomes increasingly important in recommendation modeling, recent architectures have advanced the modeling of two broad sources of ranking signals along separate paths: non-sequence features, including user, item, context, and cross features; and sequence features from user behavior histories. Wukong and HSTU have emerged as representative scalable backbones for these paths: Wukong scales high-order non-sequence feature-interaction modeling, while HSTU scales long user-behavior sequence modeling. Despite their complementary strengths, practical architectures that combine these two types of feature modeling remain underexplored. We present WHALE, a scalable unified recommendation architecture that jointly models non-sequence and sequence features on top of Wukong and HSTU. Each WHALE layer contains a Wukong module, an HSTU module, and an attention-based fusion module in which Wukong-derived interaction representations query HSTU-derived behavior representations. This design keeps both backbones active throughout the network and enables progressive Wukong-HSTU exchange, allowing high-order feature crosses to repeatedly retrieve fine-grained evidence from long user histories. To make WHALE practical for industrial deployment, we introduce customized Triton kernels and other model-systems co-design techniques to improve training and inference efficiency. On large-scale industrial recommendation data, WHALE achieves consistent gains in offline experiments. Additionally, it delivers positive online gains with a modest serving-throughput trade-off. The method has been deployed in production systems. Overall, WHALE provides a practical example of how these two sources of information can be scalably unified in an industrial recommendation model.

Figures

Figures reproduced from arXiv: 2607.17017 by Dawei Sun, Jing Qian, Maggie Zhuang, Renqin Cai, Rui Li, Velvin Fu, Xuan Cao, Yuanjun Yao, Yu Shi, Zhiyong Wang, Zhongnan Fang.

Figure 1
Figure 1. Figure 1: Overview of Wukong, HSTU, a shallow-hybrid design, and the proposed WHALE architecture. Wukong provides a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Detailed overview of the WHALE architecture. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model-quality comparison between WHALE and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Scaling analysis of WHALE with respect to sequence length, model depth, and model width. In the sequence length [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 11 linked inside Pith

  1. [1]

    Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and A. Aziz. 2022. Understanding Scaling Laws for Recommendation Models.arXiv preprint arXiv:2208.08489(2022)

    Pith/arXiv arXiv 2022

  2. [2]

    Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vincent Gatto, and Ed H. Chi. 2018. Latent Cross: Making Use of Context in Recurrent Recommender Systems. InProceedings of the Eleventh ACM International Conference on Web Search and Data Mining

    2018

  3. [3]

    Fedor Borisyuk, Mingzhou Zhou, Qingquan Song, Siyu Zhu, Birjodh Tiwana, Ganesh Parameswaran, Siddharth Dangi, Lars Hertel, Qiang Charles Xiao, Xi- aochen Hou, et al . 2024. LiRank: Industrial Large Scale Ranking Models at LinkedIn. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4804–4815

    2024

  4. [4]

    Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al. 2025. LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 247–256

    2025

  5. [5]

    Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al

  6. [6]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. InProceedings of the 10th ACM Conference on Recommender Systems. 191–198

    2016

  7. [7]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InInternational Conference on Learning Representations

    2024

  8. [8]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R’e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R’e. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems

    2022

  9. [9]

    Qin Ding, Kevin Course, Linjian Ma, Jianhui Sun, Ruochen Liu, Zhao Zhu, Chunx- ing Yin, Wei Li, Dai Li, Yu Shi, Xuan Cao, Ze Yang, Han Li, Xing Liu, Bi Xue, Hongwei Li, Rui Jian, Daisy Shi He, Jing Qian, Matt Ma, Qunshu Zhang, and Rui Li. 2026. Bending the Scaling Law Curve in Large-Scale Recommendation Systems.arXiv preprint arXiv:2602.16986(2026). doi:1...

    doi:10.48550/arxiv.2602.16986 2026

  10. [10]

    van de Geijn

    Kazushige Goto and Robert A. van de Geijn. 2008. Anatomy of High-Performance Matrix Multiplication.ACM Trans. Math. Software34, 3 (2008), 1–25. doi:10.1145/ 1356052.1356053

    arXiv 2008

  11. [11]

    Audrunas Gruslys, Remi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves

  12. [12]

    Huan Gui, Ruoxi Wang, Ke Yin, Long Jin, Maciej Kula, Taibai Xu, Lichan Hong, and Ed H. Chi. 2023. Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems.arXiv preprint arXiv:2311.05884 (2023)

    Pith/arXiv arXiv 2023

  13. [13]

    InAdvances in Neural Information Processing Systems 29

    Memory-Efficient Backpropagation Through Time. InAdvances in Neural Information Processing Systems 29

  14. [14]

    Liang Guo, Wei Li, Lucy Liao, Huihui Cheng, Rui Zhang, Yu Shi, Yueming Wang, Yanzun Huang, Keke Zhai, Pengchao Wang, Timothy Shi, Xuan Cao, Shengzhi Wang, Renqin Cai, Zhaojie Gong, Omkar Vichare, Rui Jian, Leon Gao, Shiyan Deng, Xingyu Liu, Xiong Zhang, Fu Li, Wenlei Xie, Bin Wen, Rui Li, Lu Fang, Xing Liu, and Jiaqi Zhai. 2026. Request-Only Optimization ...

    arXiv 2026

  15. [15]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. arXiv preprint arXiv:1703.04247(2017)

    Pith/arXiv arXiv 2017

  16. [16]

    Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, An- toine Atallah, Stuart Bowers, and Joaquin Quiñonero Candela. 2014. Prac- tical Lessons from Predicting Clicks on Ads at Facebook. InProceedings of the Eighth International Workshop on Data Mining for Online Advertising. 1–9. doi:10.1145/2648584.2648589

    arXiv 2014

  17. [17]

    Mingming Ha, Guanchen Wang, Linxun Chen, Xuan Rao, Yuexin Shi, Tianbao Ma, Zhaojie Liu, Yunqian Fan, Zilong Lu, Yanan Niu, Han Li, and Kun Gai. 2026. UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems. arXiv preprint arXiv:2604.00590(2026)

    arXiv 2026

  18. [18]

    Yunwen Huang, Shiyong Hong, Xijun Xiao, Jinqiu Jin, Xuanyuan Luo, Zhe Wang, Zheng Chai, Shikang Wu, Yuchao Zheng, and Jingjian Lin. 2026. Hy- Former: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction.arXiv preprint arXiv:2601.12681(2026). arXiv:2601.12681 https://arxiv.org/abs/2601.12681

    arXiv 2026

  19. [19]

    Bojian Hou, Xiaolong Liu, Xiaoyi Liu, Jiaqi Xu, et al . 2026. Kunlun: Establish- ing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design.arXiv preprint arXiv:2602.10016(2026)

    Pith/arXiv arXiv 2026

  20. [20]

    Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. doi:10.1145/ 3219819.3220023

    arXiv 2018

  21. [21]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361(2020)

    Pith/arXiv arXiv 2020

  22. [22]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherni- avskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kon- dratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao,...

    Pith/arXiv arXiv 2019

  23. [23]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. InInternational Confer- ence on Learning Representations

    2018

  24. [24]

    PyTorch Contributors. 2025. torch.cuda.set_sync_debug_mode. PyTorch Docu- mentation. Documents PyTorch’s debug mode for CUDA synchronizing opera- tions. Accessed May 10, 2026. https://docs.pytorch.org/docs/2.9/generated/torch. cuda.set_sync_debug_mode.html

    2025

  25. [25]

    NVIDIA. 2026. Matrix Multiplication Background User’s Guide. NVIDIA Doc- umentation. Accessed May 10, 2026. https://docs.nvidia.com/deeplearning/ performance/dl-performance-matrix-multiplication/index.html

    2026

  26. [26]

    PyTorch Team. 2026. Torch Compiler: AOTInductor. https://docs.pytorch.org/ docs/2.12/user_guide/torch_compiler/torch.compiler_aot_inductor.html. Ac- cessed: 2026-05-21

    2026

  27. [27]

    PyTorch Contributors. 2025. torch.nonzero. PyTorch Documentation. When input is on CUDA, torch.nonzero() causes host-device synchronization. Accessed May 10, 2026. https://docs.pytorch.org/docs/stable/generated/torch.nonzero.html

    2025

  28. [28]

    Noam Shazeer. 2020. GLU Variants Improve Transformer.arXiv preprint arXiv:2002.05202(2020)

    Pith/arXiv arXiv 2020

  29. [29]

    Steffen Rendle. 2010. Factorization Machines. In2010 IEEE International Confer- ence on Data Mining. doi:10.1109/ICDM.2010.127 RecSys ’26, September 27-October 02, 2026, Minneapolis, MN, USA Cai et al

    doi:10.1109/icdm.2010.127 2010

  30. [30]

    Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: An Intermediate Lan- guage and Compiler for Tiled Neural Network Computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Program- ming Languages. Association for Computing Machinery, New York, NY, USA, 10–19

    2019

  31. [31]

    Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. AutoInt: Automatic Feature Interaction Learning via Self- Attentive Neural Networks. InProceedings of the 28th ACM International Confer- ence on Information and Knowledge Management. doi:10.1145/3357384.3357925

    arXiv 2019

  32. [32]

    Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions.arXiv preprint arXiv:1708.05123(2017). doi:10.48550/ arXiv.1708.05123

    Pith/arXiv arXiv doi:10.48550/arxiv.1708.05123 2017

  33. [33]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. InAdvances in Neural Information Processing Systems. https: //arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2017

  34. [34]

    Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Jiayuan He, Yinghai Lu, and Yu Shi. 2024. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Rese...

    2024

  35. [35]

    Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. InProceedings of the Web Conference 2021. arXiv:2008.13535 doi:10.1145/3442381.3450078

    Pith/arXiv arXiv 2021

  36. [36]

    Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. 2025. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender.arXiv preprint arXiv:2510.26104(2025)

    arXiv 2025

  37. [37]

    Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Shen Li, Yanli Zhao, Yuchen Hao, Yantao Yao, Ellie Dingqiao Wen, Jongsoo Park, Maxim Naumov, and Wenlin Chen. 2024. Wukong: Towards a Scaling Law for Large-Scale Recommen- dation. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235)...

    2024

  38. [38]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click- Through Rate Prediction. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. doi:10.1145/3219819.3219823

    arXiv 2018

  39. [39]

    Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep Interest Evolution Network for Click-Through Rate Prediction. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 5941–5948. doi:10.1609/aaai.v33i01.33015941

    doi:10.1609/aaai.v33i01.33015941 2019

  40. [41]

    Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, Huizhi Yang, Zheng Chai, Zhe Chen, Yuchao Zheng, Qiwei Chen, Feng Zhang, Xun Zhou, Peng Xu, Xiao Yang, Di Wu, and Zuotao Liu. 2025. RankMixer: Scaling Up Ranking Models in Industrial Recommenders.arXiv preprint arXiv:2507.15551(2025)

    Pith/arXiv arXiv 2025

  41. [2016]

    InProceedings of the 1st Workshop on Deep Learning for Recommender Systems

    Wide & Deep Learning for Recommender Systems. InProceedings of the 1st Workshop on Deep Learning for Recommender Systems. 7–10