{"id":"a75fae65-43eb-43d0-bf03-4a123de6d133","arxiv_id":"2412.07585","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"Transformer-based sequential recommenders exhibit power-law and saturating NDCG scaling with model size and training interactions, enabling compute-aware model selection and effective pre-train/fine-tune transfer.","lead":"This paper trains transformer-based sequential recommendation models on Amazon purchase data and measures how ranking quality improves as models and datasets grow. It reports scaling laws similar to those seen in large language models, and shows that larger pre-trained models fine-tuned on smaller domains beat training from scratch.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tables 1 and 4 compare SRT scores evaluated with 10,000 negatives against literature baselines evaluated with 100 negatives, so the claimed gains over prior methods are not established until a matched protocol is used.","rationale":"The paper's core scaling-law finding is plausible and supported by released code; the parametric fits would benefit from held-out points and confidence intervals, but they are not internally contradicted. The sharpest problem is the evaluation-protocol mismatch. Section 4.1 explicitly chooses 10K negatives and even contrasts it with SASRec's 100 negatives, yet Tables 1 and 4 compare SRT scores against literature numbers produced under the 100-negative convention. Table 1 states the cited model numbers are 'for compatibility of evaluation methodology,' which is contradicted by the paper's own protocol. This mismatch directly affects the 'outperforms all alternatives' contribution. The direction of the bias is not obvious a priori: larger negative pools usually lower NDCG for a fixed model, so SRT might actually be stronger under a matched protocol, but the conclusion cannot be read off the current tables. Because the issue is concrete and fixable, the conditional verdict is appropriate; no change to the reader's verdict is needed. If the matched-protocol experiment confirms SRT's superiority, the transfer claim is restored; if not, the claim should be downgraded.","tokens_in":19230,"tokens_out":11422,"duration_ms":103963,"concrete_test":"Using the released code and the same 5-core Beauty/Sports splits, re-evaluate SRT-1K (from scratch and fine-tuned) with both 100 and 10,000 negative samples per positive, and obtain or reimplement SASRec, BERT4Rec, ICLRec, MINCE, CoSeRec, and CL4SRec under the same two pools. Report NDCG@5 and Hit@5 for every method in each pool. If SRT-1K fine-tuned is best under the 100-negative pool as well as the 10,000-negative pool, the transfer claim is robust; if any baseline surpasses it under 100 negatives, the table's headline comparison should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 fixes the SRT evaluation to 10,000 random negatives per positive, while Table 1 cites SASRec/BERT4Rec numbers from Chen et al. [5] and Table 4 cites MINCE/ICLRec/CoSeRec/CL4SRec numbers from their original papers, which standardly use 100 negatives. The paper never states the negative-pool size of those baselines. Because NDCG@5 over 100 candidates is structurally different from NDCG@5 over 10,001 candidates—a random positive is top-5 with probability 5/101 versus 5/10001—the numbers are not commensurate. Consequently, Table 1's 'compatibility of evaluation methodology' claim and Table 4's 'outperforms all alternatives' claim are unsupported as reported. The 12%/7% relative improvements of fine-tuned over from-scratch SRT are internally consistent and not damaged by this issue, but the external comparison to prior methods is load-bearing for the transferability contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SRT, a transformer-based sequential recommender that replaces the catalog-sized item embedding table with a trainable feature extractor over tokenized item text, making the model parameter count independent of catalog size. The authors train SRT on the full Amazon Product Data (APD) and report NDCG@5 for eight model sizes and dataset sizes up to 8.2M interactions. From these runs they fit two parametric laws: a sigmoid relating NDCG@5 to log FLOPs (Eq. 3) and a risk-decomposition form NDCG(N,T) = E - A/N^alpha - B/T^beta (Eq. 5). They also fine-tune large pre-trained SRT models on the Beauty and Sports subsets and report improvements over from-scratch training and over published baselines. The paper claims that these results reveal scaling behaviors similar to language modeling and provide a practical roadmap for compute-optimal training and transfer in sequential recommendation.","tokens_in":19567,"tokens_out":4182,"duration_ms":39892,"significance":"If the proposed scaling laws are genuinely predictive, the paper would be a useful step toward compute-aware design of sequential recommender systems, and the SRT architecture itself is a reusable contribution. The release of code and models is a strength, as is the use of the full APD rather than only 5-core subsets. However, the claim that SRT outperforms prior methods rests on an evaluation protocol that is not matched to the cited baselines, and the fitted laws lack validation on held-out configurations or uncertainty quantification. The scaling-law results are therefore currently better described as descriptive curve fitting than as established predictive laws, so the central claims need additional support.","major_comments":[{"comment":"SRT is evaluated with 10,000 random negatives per positive (Sec. 4.1), while the baseline numbers in Tables 1 and 4 are taken from prior papers that standardly use 100 negatives; the paper never states the negative-pool size for those baselines. NDCG@5 computed over 10,001 candidates is not commensurable with NDCG@5 over 101 candidates, so the claims that SRT-1K 'outperforms all other alternatives' (Sec. 3) and that the fine-tuned variants outperform all alternatives by a margin (Sec. 4.5) are not supported as reported. The authors should re-run the baselines under the same negative-sampling protocol, or explicitly restrict the external comparison to NDCG values obtained with identical candidate sets.","section":"Sec. 4.1 / Tables 1 and 4"},{"comment":"The scaling law in Eq. (5) is fitted to the same runs it is then used to describe; the paper reports no held-out configurations, no cross-validation, and no bootstrap or other uncertainty estimates, and the fit is based on only eight model sizes and dataset sizes up to 8.2M interactions. A 'prediction' for a given (N,T) pair is therefore a direct evaluation of the fitted curve rather than a test of the law's validity. To support the claim that NDCG(N,T) is predictable, the authors should fit the functional form on a subset of configurations and evaluate it on held-out configurations, reporting residuals and prediction intervals.","section":"Sec. 4.4.2 / Eq. (5)"},{"comment":"The sigmoidal envelope in Eq. (3) is constructed by selecting, for each FLOP budget, the maximum NDCG among the runs in the experiment. With a small number of runs, this envelope is sensitive to the particular configurations included, and the fit parameters are again evaluated on the same points that generated them. The claimed diminishing-returns point at log(FLOPs)=30.7 is thus not validated. The authors should provide a validation procedure, such as leave-one-configuration-out fits, and report error bars on the sigmoid parameters.","section":"Sec. 4.4.1 / Eq. (3)"},{"comment":"The paper defines T as 'seen interactions' equal to the dataset size multiplied by the number of epochs, and treats multi-epoch training as equivalent to seeing more data. This equivalence is nontrivial: repeated passes over the same sequences are not independent samples, and the improvement from multiple epochs may partly reflect optimization dynamics rather than additional data diversity. If T conflates these two effects, the interpretation of Eq. (5) as a scaling law in data size is weakened. The authors should test whether the fitted exponents and the quality of the fit change when T is defined as the number of unique interactions rather than seen interactions.","section":"Sec. 4.3 / Sec. 4.4.2"}],"minor_comments":[{"comment":"The text states that log(FLOPs)=30.7 corresponds to approximately 2.15e-13 FLOPs; this should be 2.15e13 FLOPs, since e^30.7 is on the order of 10^13.","section":"Sec. 4.4.1"},{"comment":"Eq. (5) is introduced with 'N denotes the total parameter count,' but Sec. 4.3 and Figure 5 emphasize non-embedding parameters, and Figure 7's axis is labeled only 'N'. The authors should clarify which definition of N was used in the fit and whether token embeddings are included.","section":"Sec. 4.4.2"},{"comment":"The caption of the left panel says 'number of seen iterations' but should say 'number of seen interactions' to match the terminology used in the text.","section":"Figure 5"},{"comment":"The notation N is used both for the set of negatives in Eq. (2) and for the parameter count in Eq. (4); please use distinct symbols to avoid ambiguity.","section":"Sec. 4.4.2"},{"comment":"The column labels are easy to misread; please label each column explicitly as APD raw, APD 5-core, Beauty raw, Beauty 5-core, Sports raw, and Sports 5-core.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The negative-sampling mismatch is load-bearing for the transferability claim but is fixable by re-running baselines under a common protocol. The scaling-law validation is also fixable through held-out configurations and uncertainty reporting. I see no fundamental flaw that would require rejection; the paper's contribution would be substantially stronger after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is the first paper I've seen that fits scaling laws for sequential recommendation, and it deserves credit for that. The architecture is a sensible move: replace item-id embeddings with a text-based feature encoder plus sampled softmax, so parameter count decouples from catalog size. Code and models are public. The internal comparison on a matched protocol (fine-tuned vs. from-scratch SRT) shows credible 12%/7% NDCG@5 improvements on Beauty and Sports.\n\nThe soft spots are real and they are load-bearing. First, the external baselines are not evaluated on the same protocol. The paper evaluates SRT with 10,000 random negatives per positive (Sec. 4.1) but cites SASRec/BERT4Rec numbers from Chen et al. and MINCE/ICLRec/CoSeRec/CL4SRec from their original papers, which standardly use 100 negatives. NDCG@5 over a 101-candidate pool is structurally different from NDCG@5 over a 10,001-candidate pool. The paper never states the baseline negative-set size, so Tables 1 and 4 comparisons are not established. The stress-test note is correct. This is fixable by rerunning baselines on the same protocol or reporting SRT-100/300/1K numbers for comparison.\n\nSecond, the scaling-law fits are descriptive, not predictive. Eq. (3) and Eq. (5) are fit to the same points they describe; there is no held-out configuration, no bootstrap, and no error bars on the exponents. With only eight model/data configurations, the four- and five-parameter formulas are too flexible to support the 'extrapolate to novel regimes' language. The paper should either validate on a held-out configuration or soften the claim to interpolation.\n\nThird, some numeric slips. The abstract claims use of the 'full Amazon Product Data,' but the experiments use sampled subsets up to 8.2M interactions, a small fraction of the raw 82.7M. In Sec. 4.4.1, 'log(FLOPs)=30.7 corresponds to approximately 2.15e-13 FLOPs' is a decimal error; it should be around 5e30. The stated midpoint NDCG 0.0525 doesn't follow from the sigmoid equation either. These are minor but should be cleaned up.\n\nThe related-work section correctly identifies that prior scaling studies in recommendation covered CTR and general user representations, not sequential recommendation, so the novelty claim holds. Overall, a serious referee should see this; the direction is important, the base architecture is sound, and the central issues are fixable. I'd send it out.","headline":"First scaling-law analysis for sequential recommendation, with a clean catalog-independent architecture, but the quantitative claims need matched evaluation and held-out validation.","tokens_in":20008,"tokens_out":3927,"would_cite":true,"duration_ms":34336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that transformer-based sequential recommenders exhibit language-model-style scaling laws, making NDCG@5 predictable from model size and training data.","keywords":["scaling laws","sequential recommendation","transformers","transfer learning","sampled softmax","Amazon Product Data","NDCG@5","fine-tuning"],"falsifier":"Re-run MINCE, ICLRec, CoSeRec, and CL4SRec on the same Beauty and Sports splits using the same 10,000-negative NDCG@5 protocol as the SRT models; if their scores rise above 0.0405 and 0.0206, the claimed superiority of fine-tuned SRT-1K does not hold.","tokens_in":19017,"feed_emoji":"📈","tokens_out":6552,"duration_ms":57471,"temperature":0.7,"pith_summary":"Sequential recommendation models based on transformers are typically trained to classify the next item among a fixed catalog, which couples model size to the number of catalog items. This paper proposes a variant, SRT, that replaces the item-embedding table with a trainable feature extractor over item text and trains with a sampled-softmax contrastive loss, so the parameter count no longer depends on catalog size. Using the full Amazon Product Data set, the authors show that NDCG@5 follows scaling laws of the same family as language models, namely a risk-decomposition form in model size and training data, and a saturating sigmoid in compute. They also report that pre-trained larger models fine-tuned on the Beauty and Sports subsets outperform the same models trained from scratch and several prior methods. If these claims hold, recommendation quality at a fixed compute budget becomes predictable, and transferable pre-training offers a practical route to better recommenders.","feed_headline":"Sequential recommenders obey language-model scaling laws","feed_subtitle":"On the full Amazon catalog, NDCG@5 follows predictable power laws, and fine-tuning large models beats training from scratch.","key_machinery":"The central object is the Scalable Recommendation Transformer (SRT), a modification of SASRec in which each catalog item is represented not by a trainable lookup embedding but by the output of a trainable feature extractor applied to the item's title and brand tokens; this decouples the parameter count from catalog size. The learning signal is a sampled-softmax contrastive loss over a popularity-sampled set of negatives with a logQ correction, which generalizes the single-negative losses used by SASRec and BERT4Rec. The scaling analysis rests on a risk-decomposition functional form $NDCG(N,T) = E - A/N^{\\alpha} - B/T^{\\beta}$, fitted to the envelope of best-performing runs across model sizes and dataset sizes, and on FLOP counts as the compute proxy. This combination is what lets the authors separate the effects of parameter count and data volume and extrapolate performance to untested configurations.","core_discovery":"On its own terms, the paper establishes that a transformer-based sequential recommender trained on the full Amazon Product Data exhibits two scaling regularities. First, maximum NDCG@5 as a function of compute follows a saturating sigmoid with an upper limit of about 0.149, so gains from extra FLOPs diminish. Second, for fixed model complexity $N$ and number of seen interactions $T$, the achievable NDCG@5 is fit by $NDCG(N,T) = 0.163 - 18.56/N^{0.376} - 2.9/T^{0.364}$, with nearly equal exponents on $N$ and $T$, meaning parameters and data contribute similarly at the margin. The paper further claims that the scaling transfers: models pre-trained on the full data and then fine-tuned on small domains outperform both models trained from scratch and stronger published baselines, with relative NDCG@5 gains above 12% on Beauty and 7% on Sports.","pith_inferences":["Editorial inference: If the scaling laws hold beyond the Amazon catalog, the same SRT recipe could be applied to other marketplaces with different item text, making pre-training transferable across businesses; a test would be fine-tuning on an unseen catalog's small domain.","Editorial inference: The saturating sigmoid suggests an irreducible ambiguity in next-item prediction from a long-tailed catalog, which no amount of compute alone can overcome; content-side signals or better item representations would be the next lever.","Editorial inference: The paper's protocol uses 10,000 sampled negatives for its own models, whereas published baselines are cited from papers that typically use 100 negatives; re-evaluating baselines under the 10,000-negative protocol would test whether the reported margins are protocol artifacts."],"forward_implications":["For a fixed FLOPs budget, the fitted sigmoid gives an estimated ceiling for achievable NDCG@5, so practitioners can stop spending compute once the budget passes the point of diminishing returns.","Because the exponents on $N$ and $T$ are nearly equal, balanced increases in model size and training data should be roughly as effective as doubling either alone.","SRT's parameter count is independent of catalog size, so adding or removing items does not require changing the model's vocabulary or retraining the embedding table.","Fine-tuning larger pre-trained models on smaller domains improves NDCG@5 by more than 12% on Beauty and 7% on Sports relative to from-scratch training, suggesting pre-training at scale is a viable deployment strategy.","The paper's scaling laws are derived on the full Amazon data, so they can guide data collection and model selection before training large models on a new catalog."],"supporting_citations":[{"why":"Supplies the base transformer architecture (SASRec) that SRT modifies, and the random-negative evaluation convention it scales up.","marker":"[22]"},{"why":"Establishes the language-model scaling-law framework whose functional family the paper adapts to recommendation.","marker":"[23]"},{"why":"Provides the compute-optimal analysis and risk-decomposition functional form used to fit NDCG(N,T).","marker":"[17]"},{"why":"Supplies the full Amazon Product Data set on which all scaling experiments are run.","marker":"[37]"},{"why":"Provides the ICLRec baseline and the reported SASRec/BERT4Rec metrics used for compatibility in Table 1.","marker":"[5]"},{"why":"Provides the BERT4Rec baseline compared in Table 1 and related work context.","marker":"[48]"},{"why":"Motivates the sampled-softmax loss used in the contrastive training formulation.","marker":"[57]"},{"why":"Supplies the logQ correction that adjusts the sampled-softmax loss for the popularity sampling distribution.","marker":"[60]"},{"why":"Provides the MINCE baseline and the feature-based item-representation idea related to the SRT encoder.","marker":"[40]"},{"why":"Supplies the elastic weight consolidation used during fine-tuning to prevent catastrophic forgetting.","marker":"[25]"}],"fun_headline_variants":["Recommenders scale like language models on Amazon data","Full catalog reveals recommender scaling laws","Fine-tuned large recommenders beat scratch training","Sequential recommenders follow LLM-like curves","Predictable NDCG gains from compute and data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison that supports transferability assumes NDCG@5 over 10,000 random negatives is directly comparable to published baseline numbers that were obtained with 100 negatives, even though the negative-set size for the cited baselines is never stated.","fun_headline_variants_meta":{"raw":{"variants":["Recommenders scale like language models on Amazon data","Full catalog reveals recommender scaling laws","Fine-tuned large recommenders beat scratch training","Sequential recommenders follow LLM-like curves","Predictable NDCG gains from compute and data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3367,"prompt_tokens":1009,"completion_tokens":2358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2288}},"tokens_in":625,"tokens_out":2358,"duration_ms":17396,"temperature":1.0,"reasoning_tokens":2288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:41:43.044875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MINCE, ICLRec, CoSeRec, and CL4SRec on the same Beauty and Sports splits using the same 10,000-negative NDCG@5 protocol as the SRT models; if their scores rise above 0.0405 and 0.0206, the claimed superiority of fine-tuned SRT-1K does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base transformer architecture (SASRec) that SRT modifies, and the random-negative evaluation convention it scales up."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the full Amazon Product Data set on which all scaling experiments are run."},{"cited_title":"McAuley, and Caiming Xiong","cited_arxiv_id":null,"evidence_quote":"Provides the ICLRec baseline and the reported SASRec/BERT4Rec metrics used for compatibility in Table 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the logQ correction that adjusts the sampled-softmax loss for the popularity sampling distribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MINCE baseline and the feature-based item-representation idea related to the SRT encoder."}],"review_version":1}