{"id":"d92912b0-dbc3-4adc-8d3e-71bc5d33a865","arxiv_id":"2602.07298","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Recommender LLMs follow power-law scaling when continually pre-trained on layered synthetic data, but the scaling is measured on synthetic holdouts rather than real ranking performance.","lead":"The paper builds recommendation training data from synthetic product sequences and claims that recommender LLMs improve along predictable power-law curves as that data grows. A generalist might read it because it promises a first quantitative way to budget compute and data for recommender LLMs—if the curves transfer to real recommendation quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed scaling laws are fit to perplexity on synthetic holdouts from the same CF/Node2Vec generator used for training; no real-user evaluation of the CPT LLMs connects those curves to recommendation quality.","rationale":"I read the paper as attempting to show that replacing noisy logs with a layered synthetic curriculum both improves data utility (TSTR) and yields predictable LLM scaling (perplexity vs tokens). The TSTR result is real evidence for the data-utility claim, and the breadth of model sizes (0.6B–8B, plus 14B/32B in §7.3) is a genuine strength. But the scaling-law claim rests on a circularity: the held-out evaluation domains are generated by the same CF mining + Node2Vec procedure as the training data. No experiment in the paper measures the CPT LLMs' recommendation quality on real user sequences; the small-model TSTR benchmark is disconnected from the scaling fits. This is the same structural weakness the reader identifies, and my additional observation about possible test leakage in TSTR strengthens rather than replaces it. The correct disposition is conditional acceptance pending a real-data scaling evaluation of the LLMs themselves (or a clear argument that synthetic perplexity is a validated proxy).","tokens_in":41801,"tokens_out":6956,"duration_ms":73221,"concrete_test":"Evaluate the four CPT LLMs (0.6B, 1.7B, 4B, 8B) from §6 on a time-ordered held-out split of real Merrec user sessions, with the constraint that no session in the test split contributes edges to the CF graph from which synthetic UIH is generated. Measure Recall@10/100 and NDCG@10 as a function of training tokens and model size, and fit ℓ(D)=L∞+A·D^{-α} to these real-data metrics. If real-user metrics do not exhibit the same power-law behavior and the same UIH > CF > Item-Text efficiency hierarchy (or if exponents change materially), the paper's conclusion is a scaling law for the synthetic generator, not for recommendation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central assertion — robust power-law scaling for an LLM in recommendation — is supported only by perplexity on synthetic holdouts that are produced by the same generative pipeline as the training data. In §6.1, the seven evaluation domains include CF splits (both-seen/one-unseen/both-unseen) and UIH splits (full graph/OOD); the UIH OOD set is 'sampled only based on edges from CF test set,' and the Node2Vec hyperparameters that generate both train and test UIH are tuned in Appendix A. Perplexity on such sets measures how well the LLM predicts the output of the known random-walk/association-rule process, not how well it predicts real user behavior. The only real-data anchor is the TSTR experiment in §5.2, but it uses GRU4Rec/NARM/STAMP/SASRec, not the CPT LLMs, and is never connected to the exponents in Eqs. 2–5 or Table 5. Moreover, the CF graph used to synthesize UIH is mined from the same Merrec logs from which the TSTR test set is drawn; the paper does not state that test interactions were excluded from graph construction, so the +130% recall@100 could partly reflect item/edge leakage. Internal inconsistencies (Figure 2 caption α=0.63–0.99 vs Table 5 α=0.45–0.59; Figure 12 'SAE outperformed SAE') underscore that the empirical support is not yet at the level of the headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a layered synthetic-data framework for continual pre-training (CPT) of LLMs in recommendation: Layer 1 grounds item semantics and collaborative-filtering association rules, and Layer 2 generates synthetic user interaction histories (UIH) via Node2Vec random walks over the CF graph. The authors report two main results. First, standard sequential models (GRU4Rec, NARM, STAMP, SASRec) trained on this synthetic data outperform the same models trained on real Merrec logs in a Train-on-Synthetic/Test-on-Real ranking evaluation, with a headline +130% recall@100 for SASRec. Second, they claim the first robust power-law scaling laws for an LLM continually pre-trained on recommendation data, fitting ℓ(D) = L∞ + A·D^{−α} across model sizes 0.6B–8B and seven evaluation domains, and derive compute-optimal allocation advice from a joint fit ℓ = E + A·N^{−α} + B·D^{−β}. Ablations claim asymmetric transfer (CF helps UIH but not vice versa) and document overfitting when UIH data is repeated too heavily.","tokens_in":42159,"tokens_out":5728,"duration_ms":60962,"significance":"If the scaling-law claim held in the strong form stated, the paper would be a notable contribution: it would give the recommendation community a quantitative tool for data/compute planning and would elevate synthetic curriculum design to a first-class scaling lever. The paper has real strengths: large-scale experiments (163B training tokens, four model families), use of a public dataset, an unusually transparent hyperparameter sweep in Appendix A, a candid appendix discussing semantic-tokenization risks, and extensive ablations. The TSTR result, if leakage-free, is also interesting. However, the central scaling-law claim is currently supported only by perplexity on synthetic holdouts produced by the same generative pipeline as the training data; the only real-data benchmark is disconnected from the scaling curves. The manuscript needs to either add an external real-user validation of the LLM scaling laws or substantially reframe the claim.","major_comments":[{"comment":"The scaling-law evaluation is circular with respect to the headline claim. All seven evaluation domains are holdouts generated by the same CF/Node2Vec machinery used to create the training data: UIH OOD is 'sampled only based on edges from CF test set,' and the Node2Vec hyperparameters are tuned in Appendix A on synthetic fidelity metrics. Perplexity on such sets measures how well the LLM predicts the chosen generative process, not how well it recommends to real users. The only real-data anchor, TSTR in §5.2, uses GRU4Rec/NARM/STAMP/SASRec rather than the CPT LLMs and is never connected to the exponents in Table 5. Please either evaluate the CPT LLMs on real held-out user behavior (with train/test separation enforced in graph construction and item-tokenization) or explicitly reframe the contribution as scaling laws for a synthetic data generator rather than for LLM recommendation capabil","section":"§6.1, §6.2, Eqs. (2)–(5)"},{"comment":"The +130% recall@100 TSTR claim may be affected by leakage. The synthetic UIH is generated from a CF graph mined from Merrec logs, and the TSTR test set is also drawn from Merrec. The paper does not state that the test interactions were excluded from association-rule mining or CF graph construction. Filtering the test set to items present in the synthetic vocabulary (as noted in §5.2) is necessary but not sufficient. Please specify the exact exclusion protocol and, if test interactions were not excluded, rerun the TSTR comparison with held-out users/items removed from graph construction and rule mining.","section":"§5.2, Figure 1"},{"comment":"The claim of 'robust' power-law scaling is not accompanied by uncertainty estimates. No seed variance, confidence intervals, or goodness-of-fit statistics are reported for the fitted α and L∞ values, and the joint fits in Eqs. (2)–(5) use only four model sizes. The reported ranges are internally inconsistent: the Figure 2 caption states αUIH = 0.63–0.99, Table 5 reports 0.453–0.588, and the individual Figure 2 panels show values up to 0.689. Additionally, some evaluation sets are small (CF both-unseen: ~75K entries / 38M tokens), so perplexity estimates may have nontrivial variance. Please add standard errors or bootstrap intervals and reconcile the reported α ranges before describing the scaling law as robust.","section":"§6.2, Table 5, Figure 2"},{"comment":"The asymmetric-transfer ablation is confounded by data-mixture changes. The comparison 'CF+UIH' versus 'UIH-only' uses different total recommendation tokens and different mixture ratios (CF 37% + UIH 13% versus UIH 50%). The reduction in UIH L∞ from 0.95 to 0.66 could therefore be due to the change in data volume, repeats, or mixture composition rather than to a specific synergistic effect of CF. Please control for the UIH token budget or perform a matched-mixture comparison before claiming that CF provides complementary signals that improve UIH learning.","section":"§7.1, Figure 4, Table 6"}],"minor_comments":[{"comment":"The caption says 'SAE clearly outperformed SAE across all domains'; this should read 'SAE outperformed RQ-Kmeans.' The surrounding text also switches between 'Approach 1/2/3' and 'SAE/RQ-VAE/RQ-kmeans' confusingly; please align the terminology.","section":"Appendix B, Figure 12"},{"comment":"The token counts and ratios do not sum consistently: General 128B + Item-Text 2.0B + CF 7.6B + UIH 2.8B = 140.4B, not 163B, and the 'Repeats' row has only three numbers for four domains. Please clarify whether these are per-epoch token counts, effective training tokens, or something else.","section":"Table 2"},{"comment":"The reduced-UIH mixture study changes both the UIH mixture ratio and the absolute number of UIH tokens simultaneously, so the 'overfitting at ~16 repeats' interpretation is confounded. A table row reporting the actual number of unique UIH tokens seen at the onset of overfitting would help.","section":"§7.2, Table 7"},{"comment":"The abstract and introduction state α≈0.45–0.59 for UIH, while Table 5 and Figure 2 include values up to 0.689 (UIH OOD). Please harmonize these reported ranges.","section":"§1, §6.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is potentially important if the scaling-law claim is retargeted or externally validated. The central problem is that the scaling curves and the real-data TSTR experiment are disconnected, and the TSTR experiment itself has a potential leakage issue. These are fixable in revision but require substantial additional work, not just copyediting. The internal numerical inconsistencies in the reported α ranges and the Appendix B 'SAE outperformed SAE' typo suggest the manuscript was not carefully proofread; the authors should be asked to reconcile all reported numbers. I would not reject the paper, but the revision must either add a real-user LLM evaluation or substantially narrow the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a substantial empirical study with a novel layered synthetic-data curriculum and the first systematic scaling-law fits for LLM continual pre-training in recommendation. If you work in LLM-for-RecSys, you should read it. But the headline claim—robust power-law scaling for an LLM in recommendation—is not yet supported for real recommendation performance. The scaling laws are fit to perplexity on synthetic holdouts produced by the same CF/Node2Vec pipeline as the training data. The UIH OOD test is sampled from CF test edges, so it is still measuring how well the model predicts the synthetic generator's output, not real user behavior. No real-user evaluation of the CPT LLMs connects the exponents in Eqs. 2–5 to ranking quality. The TSTR experiment uses GRU4Rec, NARM, STAMP, and SASRec—not the LLMs—and is never linked to the scaling curves.\n\nWhat the paper does well: the layered data-generation framework is thoughtful and clearly described. The ablation showing asymmetric transfer from CF to UIH (L∞ 0.95 → 0.66) is a real finding. The TSTR result that small sequential models trained on synthetic data beat those trained on real data is genuinely interesting, even if +130% on recall@100 needs error bars and a check that test interactions were excluded from graph construction—the paper doesn't state that. The scaling fits themselves look consistent across model sizes and data mixes, and the mixture-ratio overfitting analysis is a nice practical touch.\n\nSoft spots: no error bars or seed variance anywhere; the α ranges differ between Figure 2's caption (0.63–0.99) and Table 5 (0.45–0.59); Figure 12's caption says “SAE clearly outperformed SAE.” These are sloppy but fixable. The deeper issue is the disconnect between the scaling-law claim and any external validation. The paper should either temper the title and abstract, or add real-user ranking evaluation at LLM scale and show that the exponents predict it.\n\nWho this is for: RecSys researchers and anyone thinking about synthetic data for LLM CPT. The paper deserves a serious referee—the work is new and the framework is worth engaging—but it needs major revision before acceptance. I'd send it to review, not desk-reject, with the expectation that the authors add external validation, error bars, and artifact release, and that they stop calling these “scaling laws for LLMs in recommendation” until the perplexity-to-task link is established.","headline":"The scaling-law headline is real but only for synthetic holdouts from the same generator; the TSTR result is promising but disconnected from the LLM curves.","tokens_in":42681,"tokens_out":1851,"would_cite":true,"duration_ms":22137,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that continually pre-training recommendation LLMs on a layered curriculum of synthetic, bias-free data produces robust power-law scaling, and that this synthetic data is a superior training signal to real user logs.","keywords":["scaling laws","recommender systems","large language models","synthetic data","continual pre-training","power law","collaborative filtering","data-centric AI"],"falsifier":"Evaluate the continually pre-trained LLM on held-out real user interaction sequences from the original logs, without filtering items to the synthetic vocabulary, and check whether perplexity on that real evaluation follows the same power law in training tokens. If real-log perplexity does not decrease predictably, the scaling law is an artifact of the synthetic generator.","tokens_in":41655,"feed_emoji":"📈","tokens_out":4479,"duration_ms":46203,"temperature":0.7,"pith_summary":"This paper argues that large language models trained for recommendation have failed to show predictable scaling not because of model design but because raw user interaction logs are too noisy and biased to learn from. To test this, the authors build a two-layer synthetic-data curriculum: textual item descriptions and mined item-co-occurrence rules teach item semantics and collaborative logic, then graph-based random walks generate synthetic user browsing sequences free of position and popularity bias. They report that standard sequential models trained on this synthetic data beat models trained on real logs by 130% on recall@100, and that LLMs from 0.6B to 8B parameters continually pre-trained on the synthetic curriculum follow the power law ℓ(D) = L∞ + A·D⁻ᵅ across seven evaluation domains. If correct, the result turns recommendation LLM development from trial-and-error into a forecastable engineering problem, where performance can be predicted from data volume and compute budgets allocated accordingly.","feed_headline":"Synthetic data yields first scaling laws for recommendation LLMs","feed_subtitle":"Clean, bias-free training logs beat real interaction data and make recommender LLM performance forecastable.","key_machinery":"The load-bearing object is the two-layer synthetic data curriculum. Layer 1 converts item text into learned semantic tokens and mines item-to-item association rules into templated sentences, teaching the model the vocabulary and pairwise logic of recommendation. Layer 2 builds a weighted graph from those mined relationships and runs biased second-order random walks over it to produce synthetic user interaction sequences that contain no notion of rank or position; the walk parameters (return probability, in-out bias, stopping threshold) are tuned for fidelity. The scaling law ℓ(D) = L∞ + A·D⁻ᵅ is then fit to perplexity curves, and a joint model-size/data fit ℓ = E + A·N⁻ᵅ + B·D⁻ᵝ is used to d","core_discovery":"The central claim is that the obstacle to scaling laws in recommendation is data, not models. Raw interaction logs carry position bias, popularity bias, exposure bias, noise, and sparsity, and these pathologies flatten scaling curves; the authors replace logs with a layered synthetic curriculum—semantic item-text pairs, mined collaborative-filtering rules, and random-walk-generated user interaction histories—and report that clean power-law behavior is restored. In continual pre-training runs from 0.6B to 8B parameters on 163B tokens, held-out perplexity on synthetic test sets follows ℓ(D) = L∞ + A·D⁻ᵅ in all seven domains, with user interaction histories showing the steepest exponents (α ≈ 0","pith_inferences":["If the scaling laws transfer to real-user metrics, synthetic-data curricula could become a standard benchmark for comparing recommendation architectures without exposing private logs.","The paper leaves open whether scaling exponents persist when the same models are evaluated on genuine user sequences; a natural test is to add a real-log perplexity track to the evaluation suite.","The asymmetric CF-to-UIH transfer suggests a general curriculum design principle: teach explicit pairwise relations before sequential behavior, which may transfer to other sequential domains such as session-based search or content feed ranking.","The random-walk generator's hyperparameters were tuned on the same metrics used in the main evaluation; independent replication with fresh interaction data would clarify whether the reported exponents are stable."],"forward_implications":["Practitioners can estimate the data volume and compute needed to reach a target recommendation-model perplexity, replacing heuristic budget planning.","Data collection for recommendation LLMs should prioritize coverage of diverse user histories over accumulating more raw logs, since user-history data is the most scaling-efficient modality.","Adding collaborative-filtering rules to the curriculum is predicted to improve sequential behavior modeling substantially, while adding UIH to CF tasks is predicted to give little benefit.","Overtraining on a small synthetic corpus degrades generalization once repeats pass roughly 16–20 epochs, with the onset depending on mixture ratio rather than model scale.","Because data-bound domains (UIH) and model-bound domains (text) have opposite compute-optimal allocations, a single global compute strategy cannot be optimal; budgets should be split by data type."],"fun_headline_variants":["Synthetic data beats real logs for recommender LLM scaling","First scaling laws for recommendation LLMs, thanks to synthetic data","Clean synthetic data yields predictable scaling in rec LLMs","Why synthetic data now drives recommender LLM scaling laws","Synthetic curriculum unlocks scaling laws for recommendation LLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The premise that the synthetic holdout sets measure real recommendation capability: the UIH out-of-distribution test set is itself sampled from collaborative-filtering edges and generated by the same random-walk machinery, so the clean power laws may capture how well the model learns the generator's statistics rather than how well it would rank items for real users.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data beats real logs for recommender LLM scaling","First scaling laws for recommendation LLMs, thanks to synthetic data","Clean synthetic data yields predictable scaling in rec LLMs","Why synthetic data now drives recommender LLM scaling laws","Synthetic curriculum unlocks scaling laws for recommendation LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1287,"prompt_tokens":761,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":505,"tokens_out":526,"duration_ms":5317,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:37:04.565319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the continually pre-trained LLM on held-out real user interaction sequences from the original logs, without filtering items to the synthetic vocabulary, and check whether perplexity on that real evaluation follows the same power law in training tokens. If real-log perplexity does not decrease predictably, the scaling law is an artifact of the synthetic generator.","supporting_citations":[],"review_version":1}