{"id":"a8c3ce16-be5c-47d6-8886-130ce216ee81","arxiv_id":"2412.00714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Deep HSTU models tend to improve recall, ranking, multi-behavior, and multi-domain performance on public data, while GPT and SASRec fail to scale, though the evidence lacks error bars.","lead":"This paper runs large generative recommendation models, especially Meta's HSTU, at increasing depths and finds that HSTU performance generally improves with more layers across recall, ranking, and multi-domain tasks. The findings support the promise of LLM-style scaling for recommendation, but the experiments are small scale and have no error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT/SASRec 'no scalability' may be an optimization-collapse artifact of fixed hyperparameters, and HSTU's depth trend lacks error bars; the scaling contrast is not yet established.","rationale":"The reader's weakest assumption concerns transfer to industrial scale; that is a legitimate limitation, but the more immediate threat is internal: the evidence for the central contrast may be produced by optimization failure and single-run noise. If the contrast is an artifact, external validity is moot. I therefore partially agree with the reader: missing variance estimates were noted, but the optimization-collapse interpretation of GPT and SASRec's 'no scalability' is the sharper concern. I credit the paper for candidly reporting mixed results on side information and for making code available, though not independently verified. A conditional verdict remains appropriate: the requested re-runs and variance reporting could convert the central claim from plausible to established, and the paper's empirical contributions are substantial enough to merit further work rather than rejection.","tokens_in":30897,"tokens_out":6694,"duration_ms":63807,"concrete_test":"Re-run Table 1 for all four backbones at depths 2, 4, 8, 16, and 32 on ML-1M, ML-20M, and AMZ-Books with at least 5 seeds, reporting mean and standard deviation. Use a per-depth hyperparameter search over learning rate and warmup under a fixed training budget for each model, and record whether GPT and SASRec still collapse at depth 8 or above. Separately, test whether HSTU's 32-block AMZ-Books HR@10 remains above its 16-block value outside the noise. If the collapses persist under tuned hyperparameters and HSTU's depth gains survive the error bars, the central scaling claim stands; otherwise the contrast is an optimization artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2's conclusion that GPT and SASRec 'show no scalability' rests on Table 1, where deeper runs collapse to near-random metrics: GPT on ML-1M falls from HR@10 0.2803 at 4 blocks to 0.0353 at 8 blocks, and SASRec on ML-20M falls from 0.2781 at 2 blocks to 0.0599 at 4 blocks. Section 5.1.4 states that original implementations' hyper-parameters were kept fixed except for depth. A collapse of this magnitude is the signature of training instability under an unadjusted learning rate or warmup schedule, not of an absent scaling law. The 'origin of scaling law' experiments in Section 5.3.3 likewise rehabilitate SASRec by changing the residual pattern, which is a trainability fix; whether this should be called a scaling-law origin is therefore not settled. HSTU's own evidence is also noisy: on AMZ-Books, Table 1 gives HR@10 0.0680 at 16 blocks and 0.0584 at 32 blocks, a 14% drop, and on ML-1M the 16-to-32-block change is 0.3322 to 0.3298. With no standard deviations, seeds, or per-depth hyperparameter information, the claim that HSTU scales reliably while GPT and SASRec do not is not yet statistically supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of scaling behavior in transformer-based recommendation models, with a focus on Meta's HSTU architecture. Using public datasets (ML-1M, ML-20M, AMZ-Books, CIKM, IJCAI, AMZ-MD), the authors compare HSTU, Llama, GPT, and SASRec as the number of attention blocks grows, concluding that HSTU and Llama exhibit better scalability while GPT and SASRec do not. They then ablate HSTU components (relative attention bias, SiLU activation, feature interaction), run parameter analyses over embedding dimension, number of heads, sequence length, and depth, and test HSTU on complex user behavior (side information, multi-behavior, multi-domain) and ranking tasks. The paper claims to identify residual connection patterns and relative attention bias as the origins of the scaling law and states that it is the first to evaluate HSTU on ranking tasks. The experiments are broad but rely on single-run evaluations and fixed hyperparameters for baselines, which weakens the central comparative claims.","tokens_in":31139,"tokens_out":7189,"duration_ms":60832,"significance":"If the conclusions were robustly supported, this would be a useful and timely contribution: the paper asks an important question about whether recommendation backbones scale with depth, provides a wide range of experiments across six datasets and multiple tasks, and includes ablations that isolate HSTU-specific components. The first evaluation of HSTU on ranking tasks is also practically relevant. A particular strength is the explicit attempt to attribute scaling behavior to observable architectural choices rather than treating HSTU as a black box. The paper states that supplementary code is available on GitHub, which is a valuable reproducibility commitment. However, the central comparative claim (HSTU and Llama scale, GPT and SASRec do not) is currently undermined by optimization confounds and by the absence of uncertainty quantification, so the significance is conditional on the core experiments being reworked.","major_comments":[{"comment":"The conclusion that GPT and SASRec \"show no scalability\" conflates architectural scaling capacity with optimization stability under fixed hyperparameters. Section 5.1.4 states that original implementations' hyperparameters were kept fixed except for depth. Under this protocol, deeper GPT and SASRec runs collapse to near-random metrics (e.g., GPT on ML-1M drops from HR@10 0.2803 at 4 blocks to 0.0353 at 8 blocks; SASRec on ML-20M drops from 0.2781 at 2 blocks to 0.0599 at 4 blocks). Such a collapse is the classic signature of training instability under an unadjusted learning rate or warmup schedule, not evidence of an absent scaling law. The comparison should be repeated with per-depth hyperparameter tuning or with training-convergence diagnostics to separate architecture scaling from trainability.","section":"§5.2, Table 1"},{"comment":"The evidence that HSTU scales reliably is itself not statistically supported. Table 1 shows HSTU HR@10 on AMZ-Books falling from 0.0680 at 16 blocks to 0.0584 at 32 blocks, and on ML-1M the 16-to-32-block change is only 0.3322 to 0.3298. In the ranking task, Table 14 shows ML-20M AUC 0.7992 at 24 blocks dropping to 0.7914 at 32 blocks. Since no standard deviations, multiple seeds, or significance tests are reported anywhere in Section 5, the paper does not establish that HSTU scales more consistently than the baselines; several of the differences driving the qualitative conclusions are within a range that could easily be noise in single-run evaluations.","section":"§5.2, Table 1; §5.5.2, Table 14"},{"comment":"The claim that \"the product of the optimal number of layers (L) and embedding dimension (D) remains constant, supporting our theoretical model that size is proportional to O(LD)\" is not supported by the reported data. Reading the NDCG@10 columns for |S|=100, the apparent optimal depth is around 64 blocks for D=50 (LD=3200), 32 for D=100 (LD=3200), 24 for D=200 (LD=4800), and 32 for D=400 (LD=12800). The product is not constant, and the NDCG differences among neighboring depths are typically a few thousandths, so the notion of a well-defined optimum is fragile. This section should quantify the claimed relationship or reframe it as a qualitative observation.","section":"§5.3.2, Table 6"},{"comment":"The attribution of HSTU's scaling law to the residual connection pattern and relative attention bias is overreaching. The experiments modify SASRec by adopting the residual structure of HSTU or Llama and adding relative attention bias; these are precisely the architectural changes known to improve trainability of deep transformers, so the results are consistent with an optimization-stability explanation rather than a fundamental scaling-law origin. Moreover, even the best modified SASRec reaches only HR@10 0.3182 at 32 blocks on ML-20M, well below HSTU's 0.3569 (Table 1), so the modified model does not reproduce HSTU's scaling behavior. The wording should be softened to \"factors associated with improved scaling\" unless additional experiments single out the causal mechanism.","section":"§5.3.3, Table 8"},{"comment":"The text states that increasing the negative sampling ratio \"leads to a continuous improvement in the model's performance,\" but the table is non-monotonic. For HSTU on ML-20M, AUC at ratio 0.4 (0.7952) is higher than at ratio 0.8 (0.7899) and at ratio 1.0 (0.7916); on AMZ-Books, AUC drops from 0.7338 at ratio 0.8 to 0.7037 at ratio 1.0. The claim should be revised to describe the observed pattern, or the experiment should be rerun with multiple seeds to determine whether the fluctuations are within noise.","section":"§5.5.3, Table 15"}],"minor_comments":[{"comment":"The KuaiRand-27k dataset is listed in Section 5.1.1 but never appears in any reported experiment; it should either be used or removed from the list.","section":"§5.1.1, §5.1.4"},{"comment":"The paper says that original implementations' hyperparameters were kept fixed, but it does not list the actual learning rate, schedule, batch size, warmup, or dropout for each backbone; a reproducibility appendix should report these details.","section":"§5.1.4"},{"comment":"The t-SNE visualization is used to infer that \"a well-normalized model may enhance both scalability and recall performance,\" but no quantitative measure of clustering or normalization is provided, so the conclusion is not supported by the figure alone.","section":"§5.3.4, Figure 4"},{"comment":"The notation for the ranking task is confusing: the input sequence uses b′_i for labels, but the loss uses both b″_k and b′_k; the text should clearly define the ground-truth label and the prediction (e.g., y_i and ŷ_i).","section":"§5.5.1"},{"comment":"The table caption says \"Impact of various HSTU components on scaling law\" but the row labels use the abbreviation \"r.a.b.\" without defining it in the caption or text; the abbreviation should be expanded at first use.","section":"Table 2"},{"comment":"There is a duplicated in-text citation \"[2, 40, 48, 48, 66, 68]\" for efficiency optimizations, and reference [99] has missing venue and year details; these should be corrected.","section":"§6.3 and References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad empirical study with several load-bearing claims that are not yet statistically or methodologically supported. The core scaling comparison (HSTU/Llama vs GPT/SASRec) is confounded by fixed hyperparameters, and the lack of seeds and error bars means that even the positive HSTU scaling result is fragile. I would encourage the editor to request a revision that redoes the central experiments with per-depth hyperparameter tuning and multiple seeds, and to check the novelty claim about the first ranking evaluation of HSTU against recent literature that may have appeared after the ICML 2024 HSTU paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a worthwhile benchmark-and-ablation paper, not a decisive scaling-law paper. The genuinely new material is the HSTU ranking evaluation, the multi-domain and multi-behavior results, and the demonstration that SASRec becomes scalable when you give it Llama-style residual connections and relative attention bias. The central contrast—HSTU and Llama scale, GPT and SASRec don't—is not yet established, and the paper would need more work before I'd trust it.\n\nWhat I like: the authors are honest about mixed results. Side information doesn't help much, and they say so. Multi-behavior gains are uneven, and they report that too. That candor makes me believe the positive results more than I otherwise would. The ranking experiments are a real addition: nobody had systematically looked at HSTU for ranking before, and the observations about negative sampling ratios and scoring network architecture are genuinely useful. The SASRec rehabilitation is also interesting, even if the interpretation is contested. Self-citation is appropriate here; they are reproducing the reference model, and the measurements are external benchmarks, so circularity is not a real concern.\n\nThe main problem is the 'no scalability' claim for GPT and SASRec. Section 5.1.4 says the original hyperparameters were kept fixed except depth. Under that protocol, GPT on ML-1M drops from HR@10 0.2803 at 4 blocks to 0.0353 at 8 blocks, and SASRec on ML-20M drops from 0.2781 at 2 blocks to 0.0599 at 4 blocks. That pattern is the signature of training instability, not of an absent scaling law. In the scaling-law literature, you either tune the learning rate per depth or show that you tried and it didn't help. Neither is present. The same issue undercuts the 'origin of scaling law' experiments: changing the residual pattern fixes SASRec's trainability, which tells you something about optimization, not necessarily about the intrinsic scaling law.\n\nHSTU's own trend is also noisy. On AMZ-Books, HR@10 goes 0.0680 at 16 blocks and 0.0584 at 32 blocks—a 14% drop. On ML-1M, 0.3322 to 0.3298. No error bars, no seeds, no significance tests. Several conclusions in the paper ride on differences below 0.01 NDCG. The claimed theoretical O(LD) law in Section 5.3.2 appears without derivation; that needs to be removed or properly supported.\n\nThe representativeness concern is fair but minor: public datasets with embedding sizes up to 400 and sequence length 200 are a long way from trillion-parameter industrial systems, and the paper doesn't claim to bridge that gap. I'd treat the results as suggestive for that setting, not as a demonstrated industrial scaling law.\n\nWho this is for: people working on generative recommendation architectures and HSTU specifically. It's a solid empirical contribution that deserves a serious referee, but the scaling-law claims need to be tempered and the experimental reporting needs variance and per-depth hyperparameter details. I'd send it for review with major revision expected.","headline":"Useful but uneven empirical study: the HSTU ranking evaluation is new and worth having, but the central scaling-law contrast is under-evidenced.","tokens_in":31751,"tokens_out":3792,"would_cite":true,"duration_ms":33389,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that HSTU, a transformer-based generative recommendation model, scales with depth in recall and ranking while GPT and SASRec do not, and that residual connections and relative attention bias are the source of the scaling…","keywords":["scaling law","large recommendation models","generative recommendation","HSTU","sequential recommendation","ranking task","multi-behavior modeling","multi-domain recommendation"],"falsifier":"Run the same 2-to-32 block depth sweep on a production-scale behavior log with sequence lengths above 1000 and embedding dimensions above 1000, comparing HSTU, Llama, GPT, and SASRec with and without the transplanted residual and relative-bias modules. If HSTU's recall or ranking curves plateau or decline while SASRec with the transplanted modules continues to improve, or if removing the relative attention bias no longer flattens the HSTU curve, the paper's claim about the origin and transferability of the scaling law would be falsified.","tokens_in":30647,"feed_emoji":"📈","tokens_out":11994,"duration_ms":95427,"temperature":0.7,"pith_summary":"This paper tries to establish that scaling laws for large recommendation models exist but depend on architecture: as the number of transformer blocks grows, HSTU and Llama keep improving on recall, while GPT and SASRec stall or degrade. It further claims that HSTU, the representative generative recommendation backbone, also scales in ranking tasks, and that the origin of this scaling lies in the residual-connection pattern plus a relative attention bias built on position and time-difference buckets. If these claims hold, practitioners can choose HSTU-like backbones when they expect to deepen models, and they can retrofit legacy sequential models with the same modules to restore scalability.","feed_headline":"HSTU scales with depth in recall and ranking","feed_subtitle":"HSTU and Llama improve with depth; GPT and SASRec stall. Residual connections and time-aware attention drive the gains.","key_machinery":"The central object is the HSTU block, a transformer-style recommendation transducer that substitutes SiLU for Softmax in attention weighting, adds relative position and time-difference bucket biases to attention scores, and uses a residual connection placed around the pre-normalized sublayers plus a pointwise feature-interaction layer. The paper isolates the scaling law's origin by sweeping block count, ablating one component at a time, and surgically transferring the residual and relative-bias modules into SASRec, showing that these two modules, rather than raw parameter count, are what carry scalability.","core_discovery":"The paper's central claim is that model depth produces a scaling law only for certain recommendation architectures. In controlled sweeps from 2 to 32 blocks on the public rating and review datasets, HSTU and Llama improve or hold their recall metrics, whereas GPT and SASRec degrade sharply, sometimes to near-random levels. Ablations that remove one component at a time show that the relative attention bias, especially the bucketed time-difference part, is what keeps the curve rising, and that the residual-connection pattern used by HSTU and Llama is more robust than SASRec's post-normalization residual; combining Llama-style residuals with the relative attention bias turns SASRec from non-scaling into scaling. The paper also presents the first evaluation of HSTU on ranking tasks, where it improves with depth while Llama does not, and where larger numbers of negative samples help rather than hurt, provided the model size and embedding dimension are matched to the dataset.","pith_inferences":["Going beyond the paper: if the depth-scaling curve is governed by the time-bucket attention bias rather than by parameter count, then models trained on temporally regular logs should scale faster than models on sparse, irregular logs; this can be tested by changing time-bucket granularity while holding depth fixed.","The paper reports a near-constant product of optimal layers and embedding dimension but does not fit a power-law exponent; fitting the public-data curves to a two-exponent power law could give a practical rule for choosing depth and embedding size before training.","Because GPT-style backbones fail to scale in recommendation despite scaling well in language, the missing ingredient may be recommendation-specific inductive biases; a direct test would be to add the same relative time-bias and residual pattern to a GPT-style backbone and rerun the block sweep.","The negative-sampling result suggests that generative ranking models can exploit the full negative distribution, which would change how industrial click-through-rate datasets are curated if the effect transfers to production scale."],"forward_implications":["Architecture choice determines whether adding depth helps at all: HSTU- and Llama-style backbones are the ones worth scaling vertically in recall, while GPT and SASRec backbones will not reward extra blocks.","HSTU's scaling extends beyond recall: on ranking tasks it improves with depth, so the same backbone can serve both the recall and ranking stages of a recommendation pipeline.","Legacy sequential recommenders can be made scalable by transplanting the residual connection pattern and relative attention bias, offering an upgrade path for existing systems without changing the whole architecture.","Model size must be matched to dataset size: the optimal number of blocks shrinks as embedding size grows, and larger sequence lengths justify larger models, so a fixed large model is not universally best.","Generative ranking models benefit from using more negative samples rather than aggressive subsampling, which suggests that curation for ranking should preserve negatives at scale."],"supporting_citations":[{"why":"Supplies the HSTU architecture and the trillion-parameter scaling claim that this paper re-tests at smaller scale.","marker":"[126]"},{"why":"Provides the Llama backbone whose depth-scalability is compared against HSTU in the recall sweeps.","marker":"[92]"},{"why":"Provides the GPT-style backbone that the paper shows does not scale in recommendation tasks.","marker":"[1]"},{"why":"Provides the SASRec sequential baseline whose poor depth scalability motivates the residual and relative-bias transplants.","marker":"[41]"},{"why":"Supplies the two public rating datasets used for the recall and ranking depth sweeps.","marker":"[27]"},{"why":"Supplies the books and multi-domain review datasets used across the evaluations.","marker":"[62]"},{"why":"Provides the Wukong feature-interaction scaling-law model that frames the paper's comparison of scaling approaches.","marker":"[128]"},{"why":"Supplies the MBSTR multi-behavior baseline used in the complex behavior experiments.","marker":"[124]"},{"why":"Supplies the C2DSR cross-domain baseline used in the multi-domain evaluation.","marker":"[4]"},{"why":"Supplies the multi-behavior dataset used in the complex user behavior modeling experiments.","marker":"[90]"}],"fun_headline_variants":["Depth scaling depends on architecture: HSTU and Llama win","HSTU and Llama scale with depth; GPT and SASRec stall","Only HSTU and Llama benefit from depth in recommenders","First ranking eval: HSTU scales with depth, Llama doesn't","Residuals and time attention unlock depth scaling in recommenders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scaling trends measured on public datasets with embedding sizes between 50 and 400 and sequence lengths up to 200 transfer to trillion-parameter production systems with much longer user sequences; the paper never validates on a production-scale model or dataset.","fun_headline_variants_meta":{"raw":{"variants":["Depth scaling depends on architecture: HSTU and Llama win","HSTU and Llama scale with depth; GPT and SASRec stall","Only HSTU and Llama benefit from depth in recommenders","First ranking eval: HSTU scales with depth, Llama doesn't","Residuals and time attention unlock depth scaling in recommenders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001253,"raw_usage":{"total_tokens":5157,"prompt_tokens":985,"completion_tokens":4172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":4078}},"tokens_in":601,"tokens_out":4172,"duration_ms":27260,"temperature":1.0,"reasoning_tokens":4078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:05:24.957521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 2-to-32 block depth sweep on a production-scale behavior log with sequence lengths above 1000 and embedding dimensions above 1000, comparing HSTU, Llama, GPT, and SASRec with and without the transplanted residual and relative-bias modules. If HSTU's recall or ranking curves plateau or decline while SASRec with the transplanted modules continues to improve, or if removing the relative attention bias no longer flattens the HSTU curve, the paper's claim about the origin and transferability of the scaling law would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HSTU architecture and the trillion-parameter scaling claim that this paper re-tests at smaller scale."},{"cited_title":"In Proceedings of the 31st ACM International Conference on Multimedia","cited_arxiv_id":null,"evidence_quote":"Provides the Llama backbone whose depth-scalability is compared against HSTU in the recall sweeps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Wukong feature-interaction scaling-law model that frames the paper's comparison of scaling approaches."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MBSTR multi-behavior baseline used in the complex behavior experiments."}],"review_version":1}