Pith. sign in

REVIEW 4 major objections 4 minor 3 references

Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that continually pre-training recommendation LLMs on a layered curriculum of synthetic, bias-free data produces robust power-law scaling, and that this synthetic data is a superior training signal to real user logs.

desk verdict The scaling-law headline is real but only for synthetic holdouts from the same generator; the TSTR result is promising but disconnected from the LLM curves. read the letter →

arxiv 2602.07298 v3 pith:5AWDU6EM submitted 2026-02-07 cs.IR cs.AI

Benyu Zhang , Qiang Zhang , Jianpeng Cheng , Hong-You Chen , Qifei Wang , Wei Sun , Shen Li , Jia Li
show 5 more authors
Jiahao Wu Qunshu Zhang Neeraj Bhatia Xiangjun Fan Hong Yan
This is my paper · ORCID
classification cs.IRcs.AI
keywords scalinglawsrecommendersystemslargelanguagemodelssyntheticdatacontinualpre-trainingpowerlawcollaborativefilteringdata-centricAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large language models trained for recommendation have failed to show predictable scaling not because of model design but because raw user interaction logs are too noisy and biased to learn from. To test this, the authors build a two-layer synthetic-data curriculum: textual item descriptions and mined item-co-occurrence rules teach item semantics and collaborative logic, then graph-based random walks generate synthetic user browsing sequences free of position and popularity bias. They report that standard sequential models trained on this synthetic data beat models trained on real logs by 130% on recall@100, and that LLMs from 0.6B to 8B parameters continually pre-trained on the synthetic curriculum follow the power law ℓ(D) = L∞ + A·D⁻ᵅ across seven evaluation domains. If correct, the result turns recommendation LLM development from trial-and-error into a forecastable engineering problem, where performance can be predicted from data volume and compute budgets allocated accordingly.

What carries the argument

The load-bearing object is the two-layer synthetic data curriculum. Layer 1 converts item text into learned semantic tokens and mines item-to-item association rules into templated sentences, teaching the model the vocabulary and pairwise logic of recommendation. Layer 2 builds a weighted graph from those mined relationships and runs biased second-order random walks over it to produce synthetic user interaction sequences that contain no notion of rank or position; the walk parameters (return probability, in-out bias, stopping threshold) are tuned for fidelity. The scaling law ℓ(D) = L∞ + A·D⁻ᵅ is then fit to perplexity curves, and a joint model-size/data fit ℓ = E + A·N⁻ᵅ + B·D⁻ᵝ is used to d

What would settle it

Evaluate the continually pre-trained LLM on held-out real user interaction sequences from the original logs, without filtering items to the synthetic vocabulary, and check whether perplexity on that real evaluation follows the same power law in training tokens. If real-log perplexity does not decrease predictably, the scaling law is an artifact of the synthetic generator.

Watch

Extended reading notes

Core claim

The central claim is that the obstacle to scaling laws in recommendation is data, not models. Raw interaction logs carry position bias, popularity bias, exposure bias, noise, and sparsity, and these pathologies flatten scaling curves; the authors replace logs with a layered synthetic curriculum—semantic item-text pairs, mined collaborative-filtering rules, and random-walk-generated user interaction histories—and report that clean power-law behavior is restored. In continual pre-training runs from 0.6B to 8B parameters on 163B tokens, held-out perplexity on synthetic test sets follows ℓ(D) = L∞ + A·D⁻ᵅ in all seven domains, with user interaction histories showing the steepest exponents (α ≈ 0

Load-bearing premise

The premise that the synthetic holdout sets measure real recommendation capability: the UIH out-of-distribution test set is itself sampled from collaborative-filtering edges and generated by the same random-walk machinery, so the clean power laws may capture how well the model learns the generator's statistics rather than how well it would rank items for real users.

Editorial extensions

If this is right

  • Practitioners can estimate the data volume and compute needed to reach a target recommendation-model perplexity, replacing heuristic budget planning.
  • Data collection for recommendation LLMs should prioritize coverage of diverse user histories over accumulating more raw logs, since user-history data is the most scaling-efficient modality.
  • Adding collaborative-filtering rules to the curriculum is predicted to improve sequential behavior modeling substantially, while adding UIH to CF tasks is predicted to give little benefit.
  • Overtraining on a small synthetic corpus degrades generalization once repeats pass roughly 16–20 epochs, with the onset depending on mixture ratio rather than model scale.
  • Because data-bound domains (UIH) and model-bound domains (text) have opposite compute-optimal allocations, a single global compute strategy cannot be optimal; budgets should be split by data type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the scaling laws transfer to real-user metrics, synthetic-data curricula could become a standard benchmark for comparing recommendation architectures without exposing private logs.
  • The paper leaves open whether scaling exponents persist when the same models are evaluated on genuine user sequences; a natural test is to add a real-log perplexity track to the evaluation suite.
  • The asymmetric CF-to-UIH transfer suggests a general curriculum design principle: teach explicit pairwise relations before sequential behavior, which may transfer to other sequential domains such as session-based search or content feed ranking.
  • The random-walk generator's hyperparameters were tuned on the same metrics used in the main evaluation; independent replication with fresh interaction data would clarify whether the reported exponents are stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a layered synthetic-data framework for continual pre-training (CPT) of LLMs in recommendation: Layer 1 grounds item semantics and collaborative-filtering association rules, and Layer 2 generates synthetic user interaction histories (UIH) via Node2Vec random walks over the CF graph. The authors report two main results. First, standard sequential models (GRU4Rec, NARM, STAMP, SASRec) trained on this synthetic data outperform the same models trained on real Merrec logs in a Train-on-Synthetic/Test-on-Real ranking evaluation, with a headline +130% recall@100 for SASRec. Second, they claim the first robust power-law scaling laws for an LLM continually pre-trained on recommendation data, fitting ℓ(D) = L∞ + A·D^{−α} across model sizes 0.6B–8B and seven evaluation domains, and derive compute-optimal allocation advice from a joint fit ℓ = E + A·N^{−α} + B·D^{−β}. Ablations claim asymmetric transfer (CF helps UIH but not vice versa) and document overfitting when UIH data is repeated too heavily.

Significance. If the scaling-law claim held in the strong form stated, the paper would be a notable contribution: it would give the recommendation community a quantitative tool for data/compute planning and would elevate synthetic curriculum design to a first-class scaling lever. The paper has real strengths: large-scale experiments (163B training tokens, four model families), use of a public dataset, an unusually transparent hyperparameter sweep in Appendix A, a candid appendix discussing semantic-tokenization risks, and extensive ablations. The TSTR result, if leakage-free, is also interesting. However, the central scaling-law claim is currently supported only by perplexity on synthetic holdouts produced by the same generative pipeline as the training data; the only real-data benchmark is disconnected from the scaling curves. The manuscript needs to either add an external real-user validation of the LLM scaling laws or substantially reframe the claim.

major comments (4)
  1. [§6.1, §6.2, Eqs. (2)–(5)] The scaling-law evaluation is circular with respect to the headline claim. All seven evaluation domains are holdouts generated by the same CF/Node2Vec machinery used to create the training data: UIH OOD is 'sampled only based on edges from CF test set,' and the Node2Vec hyperparameters are tuned in Appendix A on synthetic fidelity metrics. Perplexity on such sets measures how well the LLM predicts the chosen generative process, not how well it recommends to real users. The only real-data anchor, TSTR in §5.2, uses GRU4Rec/NARM/STAMP/SASRec rather than the CPT LLMs and is never connected to the exponents in Table 5. Please either evaluate the CPT LLMs on real held-out user behavior (with train/test separation enforced in graph construction and item-tokenization) or explicitly reframe the contribution as scaling laws for a synthetic data generator rather than for LLM recommendation capabil
  2. [§5.2, Figure 1] The +130% recall@100 TSTR claim may be affected by leakage. The synthetic UIH is generated from a CF graph mined from Merrec logs, and the TSTR test set is also drawn from Merrec. The paper does not state that the test interactions were excluded from association-rule mining or CF graph construction. Filtering the test set to items present in the synthetic vocabulary (as noted in §5.2) is necessary but not sufficient. Please specify the exact exclusion protocol and, if test interactions were not excluded, rerun the TSTR comparison with held-out users/items removed from graph construction and rule mining.
  3. [§6.2, Table 5, Figure 2] The claim of 'robust' power-law scaling is not accompanied by uncertainty estimates. No seed variance, confidence intervals, or goodness-of-fit statistics are reported for the fitted α and L∞ values, and the joint fits in Eqs. (2)–(5) use only four model sizes. The reported ranges are internally inconsistent: the Figure 2 caption states αUIH = 0.63–0.99, Table 5 reports 0.453–0.588, and the individual Figure 2 panels show values up to 0.689. Additionally, some evaluation sets are small (CF both-unseen: ~75K entries / 38M tokens), so perplexity estimates may have nontrivial variance. Please add standard errors or bootstrap intervals and reconcile the reported α ranges before describing the scaling law as robust.
  4. [§7.1, Figure 4, Table 6] The asymmetric-transfer ablation is confounded by data-mixture changes. The comparison 'CF+UIH' versus 'UIH-only' uses different total recommendation tokens and different mixture ratios (CF 37% + UIH 13% versus UIH 50%). The reduction in UIH L∞ from 0.95 to 0.66 could therefore be due to the change in data volume, repeats, or mixture composition rather than to a specific synergistic effect of CF. Please control for the UIH token budget or perform a matched-mixture comparison before claiming that CF provides complementary signals that improve UIH learning.
minor comments (4)
  1. [Appendix B, Figure 12] The caption says 'SAE clearly outperformed SAE across all domains'; this should read 'SAE outperformed RQ-Kmeans.' The surrounding text also switches between 'Approach 1/2/3' and 'SAE/RQ-VAE/RQ-kmeans' confusingly; please align the terminology.
  2. [Table 2] The token counts and ratios do not sum consistently: General 128B + Item-Text 2.0B + CF 7.6B + UIH 2.8B = 140.4B, not 163B, and the 'Repeats' row has only three numbers for four domains. Please clarify whether these are per-epoch token counts, effective training tokens, or something else.
  3. [§7.2, Table 7] The reduced-UIH mixture study changes both the UIH mixture ratio and the absolute number of UIH tokens simultaneously, so the 'overfitting at ~16 repeats' interpretation is confounded. A table row reporting the actual number of unique UIH tokens seen at the onset of overfitting would help.
  4. [§1, §6.2.1] The abstract and introduction state α≈0.45–0.59 for UIH, while Table 5 and Figure 2 include values up to 0.689 (UIH OOD). Please harmonize these reported ranges.

Circularity Check

1 steps flagged · score 4.0 of 10

CF-to-UIH transfer claim reduces to the data-generation dependency; the central scaling-law measurements remain empirical but are evaluated only on synthetic holdouts.

  1. self definitional [§4.3.1 (Layer 2 methodology) and §7.1 (Ablation Study)]
    "First, a graph is constructed where items are represented as nodes, and the weighted edges between them are determined by the strength of the collaborative filtering relationships mined in Layer 1. ... Including CF data alongside UIH significantly improves UIH modeling performance (Figure 4, with CF+UIH achieving the lowest asymptotic perplexity (L∞ = 0.66) on UIH evaluation sets—even outperforming the UIH-only baseline (L∞ = 0.95)."

    UIH sequences are defined as random walks over a graph whose edge weights are exactly the CF association rules from Layer 1. A model trained on CF data therefore has already seen the transition probabilities of the UIH generator. The reported CF→UIH transfer is a direct consequence of the data-generation dependency, not an independent discovery of complementary signals. Presenting this as evidence that the layered curriculum provides 'complementary, non-redundant learning signals' restates the design choice rather than testing it.

full rationale

The paper's central scaling-law claim (§6.2) is an empirical fit of ℓ(D)=L∞+AD^{-α} to perplexity on held-out synthetic test sets. These test sets are generated by the same Node2Vec/association-rule pipeline as the training data, which is a standard held-out evaluation design; it limits external validity (the curves describe learning the synthetic generator, not real-user ranking) but is not by itself circular. The TSTR experiment (§5.2) provides an external real-data anchor for synthetic data utility, but it uses only small sequential models and is never connected to the LLM scaling exponents, so that gap is a correctness/validity concern rather than circularity. The one genuinely circular step is the ablation labeled 'asymmetric transfer': because UIH is generated from CF edge weights, the observation that CF data improves UIH perplexity is essentially built into the data construction. This affects one of the paper's four listed contributions but not the scaling-law fits themselves, which are internally consistent measurements. Internal inconsistencies (e.g., Figure 2 caption α=0.63–0.99 vs Table 5 α=0.45–0.59; Figure 12 'SAE outperformed SAE') are also correctness issues, not circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central results rest on hand-chosen data mixture ratios and Node2Vec hyperparameters, on the assumption that power-law forms hold, and on the assumption that synthetically generated CF/UIH perplexity measures recommendation quality. The scaling-law coefficients themselves are fitted outputs, but they inherit these choices. No new physical entity is introduced; the semantic tokens are a representation rather than a discovered object.

free parameters (2)
  • Node2Vec UIH generation hyperparameters = p=0.5, q=2.0; alpha_stop=0.15; path_conf_threshold=1e-09
    Selected by sweeping 14 configurations in Appendix A on Gini/lift and sequence metrics. These directly shape the UIH data, which is the domain with the largest reported scaling exponent.
  • Data mixture ratios = General 50%, Item-Text 9%, CF 30%, UIH 11% (full); ablated UIH ratios 0.5-15%
    Hand-set in §6.1/Table 2 and ablated in §7.2. The fitted L∞ and α change materially with these ratios (e.g. UIH L∞ ranges 0.75-0.90 in Table 7), so the headline exponents are conditional on this choice.
assumptions (5)
  • domain assumption Scaling-law functional form ℓ(D)=L∞+AD^{-α} and joint additive form ℓ=E+AN^{-α}+BD^{-β} hold for all modalities.
    Section 6.2.2 Eqs (2)-(5). The power-law form is asserted and fit, not derived from a mechanism.
  • ad hoc to paper Association rules mined from biased raw logs encode true collaborative preference once averaged over users.
    Section 4.2.2. No causal or identification argument shows that popularity/exposure bias is removed by mining co-occurrence rules from the same logs.
  • ad hoc to paper 2nd-order Node2Vec random walks over the CF graph produce unbiased user journeys.
    Section 4.3.1. The walks have no explicit position bias, but the graph weights are inherited from raw-log CF statistics, so popularity and exposure signals can propagate.
  • domain assumption Perplexity on synthetic CF/UIH holdout sets is a valid proxy for recommendation capability.
    Section 6.1 evaluation metric. No LLM-level ranking or downstream task evaluation is connected to the scaling-law curves.
  • domain assumption Qwen3 checkpoints and Merrec logs are valid base model and source data for this study.
    Section 6.1. Standard practice, but the choice affects all absolute losses and fitted constants.
invented entities (1)
  • <RECTOKEN> semantic ID tokens from a pretrained sparse autoencoder
    purpose: Item representation for LLM input, replacing raw IDs with semantic tokens.
    An internal vocabulary with no external falsifiable handle; its quality is evaluated only through the paper's own downstream and scaling metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation." pith.science (2026). https://pith.science/paper/5AWDU6EM

@misc{pith2026260207298,
  author       = {Pith},
  title        = {Pith review of: Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5AWDU6EM}},
  note         = {Machine review of arXiv:2602.07298}
}
abstract

Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are crucial for guiding research and optimizing resource allocation. We hypothesize that this may be attributed to the inherent noise, bias, and incompleteness of raw user interaction data in prior continual pre-training (CPT) efforts. This paper introduces a novel, layered framework for generating high-quality synthetic data that circumvents such issues by creating a curated, pedagogical curriculum for the LLM. We provide powerful, direct evidence for the utility of our curriculum by showing that standard sequential models trained on our principled synthetic data significantly outperform ($+130\%$ on recall@100 for SasRec) models trained on real data in downstream ranking tasks, demonstrating its superiority for learning generalizable user preference patterns. Building on this, we empirically demonstrate, for the first time, robust power-law scaling for an LLM that is continually pre-trained on our high-quality, recommendation-specific data. Our experiments reveal consistent and predictable perplexity reduction across multiple synthetic data modalities. These findings establish a foundational methodology for reliable scaling LLM capabilities in the recommendation domain, thereby shifting the research focus from mitigating data deficiencies to leveraging high-quality, structured information.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 linked inside Pith

  1. [2]

    doi: 10.1109/tkde.2025.3552658.http://dx.doi.org/10.1109/TKDE.2025.3552658

    ISSN 2326-3865. doi: 10.1109/tkde.2025.3552658.http://dx.doi.org/10.1109/TKDE.2025.3552658. Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models, 2024. https://arxiv.org/abs/2403.08763. Feiyang Kang,...

    arXiv 2025

  2. [3]

    foundation

    path_conf_threshold: A minimum cumulative path confidence threshold used to filter out low-quality sequences. B Methodology for Semantic Tokenization This appendix details the decision-making framework and methodology for determining the semantic item representation (<RECTOKEN>). As this representation serves as the fundamental vocabulary for our LLM, its...

  3. [2025]

    Just Accepted

    doi: 10.1145/3771279.https://doi.org/10.1145/3771279. Just Accepted. Athanasios Angelakis and Andrey Rass. A data-centric approach to class-specific bias in image data augmentation, 2024.https://arxiv.org/abs/2403.04120. Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus: A high-quality dataset for trainin...

    arXiv 2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.