{"id":"4505f9b4-1c93-4748-ae0b-468846d1e1bb","arxiv_id":"2506.03699","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Generative pretraining plus sparse-embedding freezing makes large Transformer ranking models scale consistently, following a power law from 13K to 0.3B dense parameters.","lead":"This paper shows that a two-step recipe, generative pretraining on user click sequences followed by freezing the learned item embeddings during discriminative fine-tuning, reduces overfitting and unlocks consistent scaling for Transformer-based CTR and CVR models. The authors report large gains on industrial and public benchmarks plus positive online A/B results, positioning the approach as a path to apply language-model scaling techniques to recommendation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The one-epoch training budget for CTR-XL makes the reported power law a statement about a fixed compute schedule, not a validated scaling law; the paper does not show the scaling curve is stable across epoch counts or that one epoch is comparable across model sizes.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern, and I agree it is the most consequential weakness in the central claim. The abstract and conclusion both assert that performance 'closely adheres to power laws' as dense parameters scale from 13K to 0.3B; every point in Figure 5 comes from the one-epoch CTR-XL budget. Because the paper itself shows both one-epoch overfitting and within-one-epoch overfitting (Section 3.3), and because the non-XL CTR task is trained for 3 epochs, the one-epoch choice is not neutral. The apparent power law could be an interaction between model scale and an arbitrary training budget rather than a capacity-limited scaling law. This does not invalidate the broader GPSD contribution: Table 3 shows consistent gains from the sparse-freeze strategies on the 3-epoch tasks, the public-dataset results are plausible, and the online A/B gains are practically meaningful. The concern is addressable with modest additional analysis of already-collected training curves, so it strengthens the condition under which the paper should be accepted rather than demanding rejection. I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":16489,"tokens_out":7900,"duration_ms":91682,"concrete_test":"Reuse the existing CTR training curves for the ST&SF strategy in Figure 4d: for L4H32A4 through L4H256A4, extract validation AUC at the end of epoch 1 and at the end of epoch 3, then fit the same power-law form to both sets of points. If the one-epoch scaling curve is not approximately parallel to the three-epoch curve, or if the smaller models improve more with additional epochs, the CTR-XL one-epoch power law is not robust. As a stronger follow-up, rerun the two endpoints of the CTR-XL study (L1H32A4 and L24H1024A16) for two or three epochs under otherwise identical settings and check whether the relative ordering and the fitted exponent change materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The scaling-law claim in Section 3.6 and Figure 5 is derived entirely from models trained for exactly one epoch on CTR-XL, as the paper states: 'we train each model for one epoch due to resource constrain.' No multi-epoch comparison is reported for this dataset, so the fitted power law and its derived bounds (AUC upper bound ≈ 0.7097, loss lower bound ≈ 0.3695) may be properties of the training budget rather than of the model family. The concern is not aesthetic: the paper itself documents one-epoch overfitting and trains the smaller CTR task for 3 epochs in Section 3.2, so a single epoch is not established as a natural or sufficient operating point across all model scales. If smaller models would gain more from additional epochs, or larger models are under-trained at one epoch, the monotone scaling curve and its asymptotic bound would be an artifact of a fixed and perhaps unlucky training schedule rather than evidence that generative pretraining with sparse freezing unlocks a genuine scaling law for discriminative recommendation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GPSD, a two-stage framework for discriminative recommendation (CTR/CVR/CART). A Transformer is first trained generatively with sampled softmax on user behavior sequences; the learned parameters are then transferred to a discriminative ranking model, with the sparse embedding parameters frozen (the ST&SF strategy being preferred). Experiments cover three industrial tasks, an expanded 5B-sample CTR-XL dataset, three public datasets, cross-architecture transfer to HSTU and Wukong, and a 30-day online A/B test. The headline claim is that, with this initialization and freezing scheme, Transformer ranking models scale consistently in dense parameters from 13K to 0.3B following a power law (AUC asymptote 0.7097, loss asymptote 0.3695), in contrast to from-scratch discriminative training, which overfits and does not scale.","tokens_in":16661,"tokens_out":5439,"duration_ms":56941,"significance":"If the scaling-law claim holds, the paper would be an important step toward making Transformer ranking models as scalable as language models, and the sparse-freeze transfer recipe is simple and likely to be adopted. The paper's strengths are the breadth of evidence: three industrial datasets plus a 5B-sample dataset, public benchmarks, cross-architecture transfer, an online A/B test, and released code. The observation that freezing transferred embeddings removes both one-epoch and within-one-epoch overfitting is interesting in its own right. However, the scaling-law evidence is currently descriptive rather than validated predictive scaling, and a confound between dense and sparse parameter growth weakens the causal reading of the central claim.","major_comments":[{"comment":"The scaling law is derived entirely from models trained for exactly one epoch on CTR-XL, as stated in §3.6 ('we train each model for one epoch due to resource constrain') and in Table 2. A scaling law that is claimed to characterize the model family should be approximately invariant to the training budget, at least in the ordering of models and in the located asymptote. The paper provides no multi-epoch results on CTR-XL for any model size. Because §3.3 and Figure 4 show that validation AUC is still changing at epoch boundaries and that generative training does not overfit, one epoch is not established as a canonical or sufficient operating point across all scales. If smaller models catch up with additional epochs, the fitted power law and the derived bounds (AUC ≈ 0.7097, loss ≈ 0.3695) would be artifacts of a fixed training schedule rather than evidence of a genuine scaling law. Please report multi-epoch runs for at least the smaller model sizes, or otherwise show that the one-epoch curves already place the models at the same relative positions as at convergence. In addition, the power law is fitted to all eight points in Figure 5; a stronger claim requires fitting on the first k sizes and predicting the remaining ones, as is standard in scaling-law validation.","section":"§3.6, Figure 5, Table 8; also §3.2"},{"comment":"The scaling experiment varies dense and sparse parameters simultaneously. Table 5 shows #Sparse Params growing from 125M to 4B while #Dense Params grows from 13K to 327M, with both tied to model width. Because the method's key ingredient is the frozen sparse embedding table, larger models also receive larger and higher-capacity embeddings. The central claim that performance scales with 'dense parameters' is therefore not identified as an effect of dense parameters alone. Please hold the sparse parameter count fixed while varying dense parameters (or decouple embedding width from Transformer width), or at least present an analysis that separates the contribution of embedding width from the contribution of Transformer depth and width.","section":"§3.6, Table 5"},{"comment":"No uncertainty quantification is provided for any of the central empirical results. Public dataset results in Table 6 are single AUC values without standard deviations over seeds; the online A/B gains in Table 7 are point estimates without confidence intervals or significance tests; and the scaling-curve points in Figure 5 are single runs. Given that some reported improvements are modest (e.g., DeepFM + ST&SF on Taobao, +5.56%), this makes it difficult to assess whether the observed orderings and gains are stable. Please add repeated-seed results for the public datasets and at least confidence intervals or significance tests for the online metrics.","section":"Table 6, Table 7, Figure 5"}],"minor_comments":[{"comment":"The power-law equations are rendered with '□' placeholders instead of minus signs (e.g., 'y = 0.7097 □ 0.7553x□0.2364'); please fix the typography.","section":"Figure 5"},{"comment":"There are several typos: 'perfromance' (§3.8), 'Addtionally' (§4.3), 'new comming' (§3.9), and 'phenomenons' (§3.3).","section":"§3.8, §4.3, §3.9, §3.3"},{"comment":"The caption says 'The underline denote the best performance' and 'bold denote the best'; these should be 'The underline denotes' and 'bold denotes'.","section":"Table 3 caption"},{"comment":"The notation LuHvAw is used but never formally defined in the text; please define it explicitly, even though the meaning is inferable from Table 5.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is substantial and likely publishable, but the headline scaling-law claim needs stronger evidence than a one-epoch fit to the same data. The one-epoch issue is especially important because the same CTR-XL dataset is used both to fit the power law and to derive the asymptotic bounds; I would require a multi-epoch or held-out scaling check before accepting the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth your time. The core idea is simple: pretrain a Transformer generatively on click sequences, then when you fine-tune on a discriminative CTR/CVR task, freeze the sparse embedding table and only train the dense layers. On industrial and public data, that consistently beats training from scratch and also beats full transfer. The online A/B lift (+7% GMV, +3.8% CTR over 30 days) is credible for a small deployed model. The overfitting analysis is also genuinely useful: they separate one-epoch overfitting from a subtler within-one-epoch overfitting, and show that generative pretraining plus sparse freezing narrows both. That is a concrete, reproducible recipe, and the public-dataset gains are large enough that the trick is not just an artifact of Alibaba's infrastructure.\n\nThe main soft spot is the scaling-law claim. It is fitted to models trained for exactly one epoch on CTR-XL because of resource constraints, and the paper does not show that one epoch is a comparable operating point across model sizes or that the curve is stable with more training. So the fitted power law and its asymptotic bound (AUC ~0.7097) are statements about a fixed training schedule, not a validated scaling law. The paper is honest about the one-epoch choice, but the claim still goes beyond the evidence. Also, there are no error bars or significance tests anywhere, the public dataset split is not exactly documented, and the code link is unverified. Those are addressable, and they do not undercut the main empirical finding: freezing sparse embeddings during discriminative fine-tuning is a real and effective fix for overfitting.\n\nI disagree with the stress-test note if it implies the scaling law is load-bearing. The method's value does not depend on the power law being a true scaling law; it depends on the transfer-plus-freeze strategy working, which the ablations and online test support. But the note is right that the scaling-law headline is overclaimed as a result.\n\nWho should read it: anyone building industrial ranking models, and anyone working on pretraining for recommender systems. The overfitting taxonomy and the freezing strategy are immediately useful. The scaling-law part is suggestive and worth investigating further, but not proof.\n\nRecommendation: yes, send this to peer review. It is a serious, well-executed paper with a useful contribution and an honest limitations section. A good referee should ask for error bars, a multi-epoch sensitivity check for the scaling curve, and a proper public-data split description, but those are revision requests, not grounds for rejection.","headline":"A practical, industry-grounded paper whose sparse-freezing trick clearly works; the scaling-law claim is real but under-verified because it rests on single-epoch training.","tokens_in":17234,"tokens_out":1321,"would_cite":true,"duration_ms":17120,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative-pretraining step that freezes sparse embeddings turns overfitting ranking Transformers into predictably scalable ones.","keywords":["recommender systems","generative pretraining","discriminative recommendation","CTR prediction","scaling law","Transformer","sparse parameter freezing","overfitting"],"falsifier":"Train the smaller models in the scaling sweep, such as L1H32A4 and L4H32A4, on CTR-XL for two or three epochs under the same ST&SF strategy; if their AUC rises to meet or exceed the largest model's 0.7018, the fitted power law is an artifact of uniform under-training rather than a genuine capacity scaling law.","tokens_in":16237,"feed_emoji":"📈","tokens_out":7615,"duration_ms":82705,"temperature":0.7,"pith_summary":"Generative pretraining combined with a frozen embedding table solves an overfitting problem that has kept Transformer ranking models small, the paper argues. Discriminative CTR/CVR models trained from scratch show two distinct overfitting modes, one at epoch boundaries and one within the first epoch, so larger models can validate worse than smaller ones. GPSD first pretrains a Transformer to predict the next item in user behavior sequences, then transfers the learned sparse embedding parameters into a discriminative ranking model and freezes them while training only dense parameters. With this recipe, the generalization gap narrows, and as dense parameters grow from 13K to 0.3B, AUC improves steadily and fits a power law with an empirical ceiling near 0.7097 on a 5-billion-sample industrial dataset. If the claim holds, ranking Transformers can be scaled predictably in the same way language models are.","feed_headline":"Frozen embeddings make ranking Transformers scale predictably","feed_subtitle":"Freezing pretrained sparse embeddings lets ranking models improve with size, following a power law from 13K to 0.3B parameters.","key_machinery":"The machinery is the GPSD framework with its Sparse Transfer & Sparse Freeze bridging strategy: a unidirectional Transformer is pretrained autoregressively to predict the next item in user behavior sequences using sampled softmax over a vocabulary of millions of item IDs, with input and output embeddings tied. The learned embedding table is then loaded into a discriminative ranking Transformer and frozen, while only the dense Transformer and MLP parameters are updated with cross-entropy loss. This separation of sparse and dense parameters is what lets generative pretraining do its work: the frozen embeddings carry stable user-item representations, and the trainable dense parameters are free to scale without the one-epoch and within-one-epoch overfitting that otherwise appear.","core_discovery":"The central claim is that the bottleneck in scaling discriminative recommendation Transformers is not the Transformer itself but the sparse embedding parameters, which overfit when trained discriminatively on sparse user behavior data. The paper shows that generative pretraining with heavy random negative sampling trains these embeddings stably, and that transferring them into a discriminative model and freezing them removes the overfitting that blocks scaling. Under this Sparse Transfer & Sparse Freeze strategy, the dense Transformer parameters scale monotonically from 13K to 327M parameters on CTR-XL, with AUC rising from 0.6306 to 0.7018 and loss falling from 0.3922 to 0.3732, both closely following fitted power laws. Merely transferring all parameters without freezing, as in standard language-model fine-tuning, does not fix the problem, which the paper takes as evidence that sparse-parameter learning is the core obstacle.","pith_inferences":["Because every point on the scaling curve was trained for exactly one epoch, the power law may reflect per-epoch learning behavior rather than true capacity; testing small models with two or three epochs would show whether the fitted curve is a real capacity law or an artifact of uniform under-training.","The empirical AUC ceiling near 0.7097 comes from scaling dense parameters alone, so further gains on large ranking datasets may require unfreezing a subset of sparse parameters, adding features, lengthening sequences, or improving data quality, directions the paper itself leaves open.","The success of cross-architecture transfer suggests the generatively pretrained embedding table encodes reusable user-interest structure independent of the dense architecture, which could be tested by attaching the same frozen embeddings to a variety of dense heads and measuring how much performance transfers.","If the sparse-freeze principle generalizes, it may change how recommendation models are pretrained: instead of matching architectures exactly, practitioners could invest in high-quality generative sparse parameters once and share them across many ranking models."],"forward_implications":["Transformer-based ranking models can be scaled predictably like language models, with AUC and loss following fitted power laws in dense parameter count, instead of plateauing or degrading at larger sizes.","Pretrained frozen sparse embeddings behave as portable assets: the paper shows they improve scalability even when transferred to non-Transformer architectures such as HSTU and Wukong.","Even a small ranking Transformer, 3 layers wide by 160 hidden units, delivers large online gains under incremental training, with reported increases of 7.03% in GMV, 2.11% in orders, 1.86% in buyers, and 3.78% in CTR.","The Sparse Transfer & Sparse Freeze recipe also improves existing non-Transformer ranking models on public datasets, with relative AUC gains between 2.36% and 17.31%, so the benefit is not limited to Transformer-only pipelines.","The fitted power law gives practitioners a way to forecast the payoff of a larger dense model before spending GPU resources, such as the empirical AUC ceiling near 0.7097 on CTR-XL."],"supporting_citations":[{"why":"Supplies the language-model scaling-law methodology that the paper adopts to fit AUC and loss as power laws of dense parameter count.","marker":"[15]"},{"why":"Documents one-epoch overfitting in CTR models and attributes it to feature sparsity, the phenomenon the paper extends to within-one-epoch overfitting on industrial data.","marker":"[40]"},{"why":"Justifies the sampled-softmax loss used for generative pretraining over a vocabulary of millions of items.","marker":"[13, 30]"},{"why":"Provides evidence that parameter scaling in recommendation models had stalled, the contrast the paper seeks to overturn.","marker":"[1]"},{"why":"Identifies embedding collapse as an obstacle to scaling recommendation models, motivating the sparse-embedding treatment at the center of GPSD.","marker":"[10]"},{"why":"Supplies the HSTU architecture used in the cross-architecture transfer experiments.","marker":"[35]"},{"why":"Supplies the Wukong architecture used in the cross-architecture transfer experiments.","marker":"[37]"},{"why":"Shows that earlier discriminative Transformer ranking models were limited to a single layer, the scalability barrier GPSD addresses.","marker":"[2]"}],"fun_headline_variants":["Generative pretraining unlocks scalable ranking Transformers","Freezing sparse embeddings lets Transformers scale in ranking","Power-law scaling for recommendation Transformers via pretraining","Sparse freeze fixes overfitting, enabling Transformer scale-up","Pretrained embeddings break ranking model scaling barrier"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scaling law is estimated from models trained for exactly one epoch on CTR-XL, because Section 3.6 says one epoch was used due to resource constraints, so the claim that larger models are genuinely better assumes one epoch is enough for every model size to reveal its true capacity.","fun_headline_variants_meta":{"raw":{"variants":["Generative pretraining unlocks scalable ranking Transformers","Freezing sparse embeddings lets Transformers scale in ranking","Power-law scaling for recommendation Transformers via pretraining","Sparse freeze fixes overfitting, enabling Transformer scale-up","Pretrained embeddings break ranking model scaling barrier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2817,"prompt_tokens":1028,"completion_tokens":1789,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1715}},"tokens_in":644,"tokens_out":1789,"duration_ms":14822,"temperature":1.0,"reasoning_tokens":1715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:57:34.566216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the smaller models in the scaling sweep, such as L1H32A4 and L4H32A4, on CTR-XL for two or three epochs under the same ST&SF strategy; if their AUC rises to meet or exceed the largest model's 0.7018, the fitted power law is an artifact of uniform under-training rather than a genuine capacity scaling law.","supporting_citations":[],"review_version":1}