Wukong: Towards a Scaling Law for Large-Scale Recommendation
read the original abstract
Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 GFLOP/example, where prior arts fall short.
This paper has not been read by Pith yet.
Forward citations
Cited by 26 Pith papers
-
The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders
Transformer recommenders amplify popularity bias via spectral collapse when scaled; SPRINT constrains attention column-sums and feed-forward spectral norms to improve fairness and scaling behavior.
-
Scaling Laws for Behavioral Foundation Models over User Event Sequences
Across 600 runs from 10^15 to 10^19 FLOPs, behavioral models show a 2% embedder is compute-optimal at all scales, training is data-heavy at low compute, and optimal negatives increase with budget until memory-limited.
-
LoopCTR: Unlocking the Loop Scaling Power for Click-Through Rate Prediction
LoopCTR trains CTR models with recursive layer reuse and process supervision so that zero-loop inference outperforms baselines on public and industrial datasets.
-
Sample Is Feature: Beyond Item-Level, Toward Sample-Level Tokens for Unified Large Recommender Models
SIF encodes full historical raw samples as tokens via hierarchical quantization to preserve sample context and unify sequential/non-sequential features in large recommender models.
-
TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds
TokenFormer unifies multi-field and sequential recommendation modeling via bottom-full-top-sliding attention and non-linear interaction representations to avoid sequential collapse and deliver state-of-the-art performance.
-
IAT: Instance-As-Token Compression for Historical User Sequence Modeling in Industrial Recommender Systems
IAT compresses each historical interaction instance into a unified embedding token via temporal-order or user-order schemes, allowing standard sequence models to learn long-range preferences with better performance an...
-
Compute Only Once: UG-Separation for Efficient Large Recommendation Models
UG-Separation framework disentangles user-side and item-side flows in TokenMixer dense-interaction models to enable reusable user computations, cutting inference latency up to 20% in ByteDance production scenarios.
-
DeRes: Decoupling Residual Stability and Adaptivity for Scalable CTR Prediction
DeRes decouples residual stability and adaptivity via identity and block-attention paths with SiLU pointwise attention, delivering up to 0.32% AUC gains and steeper scaling laws on industrial and public CTR datasets.
-
Dual-Stream MLP is All You Need for CTR Prediction
DS-MLP achieves state-of-the-art CTR prediction on three benchmarks using a final vanilla MLP structure trained via knowledge distillation and two alignment strategies.
-
FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendation
FLUID introduces LUCID semantic codes from a multimodal encoder to retire item IDs in livestreaming rankers, with staged warmup yielding online gains of +0.55% watch duration and +2.05% cold-start views.
-
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
LoKA enables practical FP8 use in numerically sensitive large recommendation models via profiling, model adaptations, and runtime kernel orchestration.
-
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
LoKA enables practical FP8 use in numerically sensitive large recommendation models via online profiling of activations, reusable model modifications for stability, and dynamic kernel dispatching.
-
Understanding DNNs in Feature Interaction Models: A Dimensional Collapse Perspective
DNNs mitigate dimensional collapse of embeddings in feature interaction models, shown via parallel and stacked experiments plus gradient analysis.
-
When Less is More: The LLM Scaling Paradox in Context Compression
Larger LLM compressors in lossy setups often yield less faithful context reconstructions due to knowledge overwriting and semantic drift, with mid-sized models outperforming larger ones across 27 tested configurations.
-
SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs
SilverTorch replaces standalone ANN indexing and filtering with a unified GPU model using a model-based Bloom index and fused Int8 ANN kernel, delivering up to 23.7x throughput and 13.35x cost efficiency gains on indu...
-
FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendation
FLUID retires candidate-side item IDs in production livestream rankers via cross-domain multimodal hierarchical codes and late-fusion ID-free design, reporting online gains of +0.55% Quality Watch Duration and +2.05% ...
-
PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents
PACEvolve++ uses a phase-adaptive reinforcement learning advisor to decouple hypothesis selection from execution in LLM-driven evolutionary search, delivering faster convergence than prior frameworks on load balancing...
-
Harmonizing Generative Retrieval and Ranking in Chain-of-Recommendation
RecoChain unifies generative candidate generation via hierarchical semantic IDs and SIM-based ranking in a single Transformer to improve top-K recommendation performance.
-
Sample Is Feature: Beyond Item-Level, Toward Sample-Level Tokens for Unified Large Recommender Models
SIF encodes entire historical raw samples as tokens via hierarchical group-adaptive quantization and token/sample-level mixing to overcome partial encoding and feature heterogeneity limits in scaled recommender models.
-
SOLARIS: Speculative Offloading of Latent-bAsed Representation for Inference Scaling
SOLARIS speculatively precomputes user-item latent representations to decouple large-model inference from real-time serving, delivering 0.67% revenue gain when deployed in Meta's ad system.
-
Beyond Dense Connectivity: Explicit Sparsity for Scalable Recommendation
SSR uses static random filters and iterative competitive sparse mechanisms to explicitly enforce sparsity in recommendation models, outperforming dense baselines on public and billion-scale industrial datasets.
-
Revisiting Content-Based Music Recommendation: Efficient Feature Aggregation from Large-Scale Music Models
TASTE dataset and MuQ-token aggregation enable effective use of audio features from large music models to improve content-based music recommendations over collaborative filtering alone.
-
CMSL: Constructive Multi-Sequence Learning for Recommendation Systems
CMSL uses a learnable module to disentangle user history into multiple pure sequences modeled with linear attention to improve recommendation performance over single-sequence approaches.
-
UniFormer: Efficient and Unified Model-Centric Scaling for Industrial Recommendation
UniFormer introduces a unified model-centric scaling approach for recommender systems via feature-space and task-space modules, semantic tokenization, and multi-sequence attention, with reported gains in production A/...
-
FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
FreeScale reduces computational bubbles by up to 90.3% in distributed training of sequence recommendation models on 256 H100 GPUs via load balancing, prioritized embedding overlap, and SM-Free communication.
-
Joint Model Parameter Scaling and Universal-Domain Data Integration for E-commerce Search Ranking
UniScale couples entire-space data construction with a hierarchical fusion transformer to improve scaling behavior and deliver 1.70% purchase and 2.04% GMV lifts in large-scale e-commerce search A/B tests.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.