Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training

Chantat Eksombatchai; Jiajing Xu; Po-Wei Wang; Yi-Ping Hsu

REVIEW 4 major objections 6 minor 15 references

Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that training ID embeddings in a separate, minimal contrastive model—before they enter a larger recommendation model—lets them train over multiple epochs without overfitting, and that this two-stage scheme lifts live engage

desk verdict A credible, practical industrial result: real A/B gains from two-stage contrastive ID pretraining, but the mechanism story is softer than the data. read the letter →

arxiv 2508.18700 v1 pith:5FPCWWTE submitted 2025-08-26 cs.IR cs.LG

Yi-Ping Hsu , Po-Wei Wang , Chantat Eksombatchai , Jiajing Xu This is my paper

classification cs.IRcs.LG

keywords recommendersystemsIDembeddingsone-epochphenomenoncontrastivepre-trainingtwo-stagetrainingoverfittinglong-taildistributiononlineA/Btest

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Online recommendation models that use ID embeddings often cannot train for more than one epoch: tail items have more free parameters than their training samples, so embeddings memorize the data and validation metrics drop. The paper proposes pre-training those shared ID embeddings separately, in a minimal dot-product model with a contrastive loss, over roughly ten times more engagement data, and then fine-tuning them inside larger downstream ranking models. It argues this two-stage scheme removes the one-epoch constraint because contrastive loss with in-batch and random negatives shrinks the effective dimensionality of tail embeddings. Offline Hit@3 results and a two-week online A/B test, which reports a 2.2% site-wide engagement gain, support the claim. The payoff if true: industrial systems can use more data and more epochs without expensive model rewrites.

What carries the argument

The mechanism is a two-stage training split. Stage 1 learns shared user-ID and item-ID embeddings in a lightweight dot-product model optimized with the contrastive loss in Eq. (1), where each positive user-item pair is contrasted with in-batch negatives plus uniform random negatives. Stage 2 loads those pretrained embeddings into downstream models and either freezes or fine-tunes them. The contrastive negatives are the load-bearing part: they lower the effective degrees of freedom of tail IDs, so extra epochs refine rather than memorize.

What would settle it

Re-run the Stage 1 pre-training comparison of Figure 2 with a strict temporal holdout and no user or item overlap between the 10x pre-training data and the evaluation set; the central claim fails if contrastive Hit@3 also peaks at epoch 1 under that protocol, or if BCE Hit@3 does not decline after epoch 1.

Watch

Extended reading notes

Core claim

The central discovery is that the one-epoch overfitting of ID embeddings is not an unavoidable property of embeddings, but a consequence of how they are trained. By pre-training a shared embedding table with a contrastive loss in a minimal dot-product model, with in-batch and uniformly sampled random negatives, the effective dimensionality of tail entries is reduced; multi-epoch training no longer degrades Hit@3. These embeddings can then be loaded into downstream homefeed and related-pins models and fine-tuned through multiple epochs, outperforming both single-stage training and frozen pretrained embeddings. In live traffic the two-stage system produced +1.11% and +1.09% engagement lifts on

Load-bearing premise

The claim that contrastive pre-training avoids the one-epoch problem rests on the Hit@3 measurements in Figure 2 and Table 1 being computed on held-out data with no leakage from the roughly 10x pre-training set; if those metrics are not properly held out, the evidence for 'no overfitting' collapses.

Editorial extensions

If this is right

ID embeddings can be pre-trained for multiple epochs and with 10x more engagement data without the validation degradation seen with BCE loss.
Downstream models should fine-tune pre-trained embeddings rather than freeze them; in the reported ablation, fine-tuning beat freezing by roughly 0.17 to 0.26 percentage points in Hit@3.
Merging the contrastive loss into a single-stage model is worse than baseline by about 3.3% and 1.9% Hit@3, so the two-stage split itself is necessary, not just the loss.
The two-stage system is deployable in web-scale traffic; the live A/B experiment reports +1.11% Homefeed and +1.09% Related Pins engagement, about 2.2% site-wide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism is general, the same two-stage recipe should transfer to other embedding-heavy tasks with long-tail IDs, such as search retrieval or graph link prediction, wherever single-epoch training is currently imposed by tail overfitting.
A testable extension the paper does not run: sweep the negative-sample mix and embedding dimension to check whether performance tracks effective dimensionality rather than the specific contrastive loss; if it does, simpler regularizers might reproduce the benefit.
The online test changed several variables at once—more data, more epochs, a new loss, and a new training split—so a decomposition study would reveal how much of the 2.2% gain comes from each component.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: this is a legitimate web-scale engineering result. The authors propose two-stage training: pre-train ID embeddings with a contrastive loss in a minimal dot-product model over 10x engagement data for multiple epochs, then fine-tune those embeddings in downstream ranking models. The specific combination, applied to the one-epoch problem, is new as far as the cited literature goes. They report a 2.2% site-wide engagement lift in live A/B tests at Pinterest. That's a real number from a production system.

What the paper does well: it is honest about the tradeoffs of industrial systems, credits the contrastive loss to Pinnerformer [8] rather than claiming it as a new invention, and Table 1 gives a useful ablation separating single-stage, frozen two-stage, and fine-tuned two-stage. The finding that two-stage fine-tuned consistently beats the baseline and the frozen variant is the core empirical takeaway. The writing is compact and direct. This is a solid short paper.

The soft spots are where the explanation goes beyond the data. The claim that contrastive loss 'reduces the effective dimensionality of tail entries' is asserted, not measured. Figure 2 shows BCE overfitting and contrastive loss generalizing, but the paper never states whether Hit@3 is computed on held-out data, and the BCE curves come from the same pretraining dataset, not from a downstream comparison of BCE-pretrained vs contrastive-pretrained embeddings. The larger issue is attribution: the two-stage system differs from the baseline in multiple ways—10x more engagement data, multi-epoch pretraining, and fine-tuning. There is no ablation holding data fixed while varying only the loss. So the 2.2% lift could plausibly come from data volume rather than the contrastive mechanism. The stress-test note is right about this underdetermination.

That said, the paper does not overreach in its conclusion; it claims the two-stage strategy helps, and the A/B test supports that whole-system claim. What is overreached is the specific mechanism. The online results also lack confidence intervals, which is common in industrial papers but worth flagging.

Who is this for: anyone building large-scale ID embedding systems or working on the one-epoch problem. It's a useful data point, not a field reorganizer. I'd send it to peer review—a serious referee can ask for held-out details and a cautious rewrite of the mechanism claim. With those, it would be a clean accept; without them, it's still a fine paper for a practitioner audience.

I'd bring it to reading group for the evaluation discussion alone.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage training strategy for ID-based embeddings in web-scale recommendation systems. Stage 1 pre-trains shared user/item ID embeddings on a lightweight dot-product model with a contrastive loss (Eq. 1) using 10x more engagement data than downstream models, over multiple epochs. Stage 2 fine-tunes these embeddings in downstream surface models. The authors claim this resolves the 'one-epoch phenomenon' because contrastive loss 'reduces the effective dimensionality of tail entries' (Section 3), and report offline Hit@3 improvements for two-stage fine-tuned embeddings (Table 1) plus a 2.2% site-wide engagement gain from live A/B tests (Table 2). The paper is a short industrial application paper from Pinterest.

Significance. If the central claim is correct, the paper identifies a practical solution to a well-known industrial problem: the one-epoch overfitting of ID embeddings in online recommendation. The proposed two-stage scheme is simple, deployable, and the reported online A/B gains are meaningful (2.2% site-wide engagement). The paper's strengths include its direct industrial validation on live traffic, a clean separation of pretraining and downstream fine-tuning, and the use of an established contrastive loss (Eq. 1, from PinnerFormer [8]) rather than a novel ad-hoc objective. The specific claim that contrastive pretraining avoids the one-epoch problem and improves downstream fine-tuning is, however, supported only by evaluations lacking held-out details and by ablations that do not isolate the contrastive loss from data-volume effects.

major comments (4)

[§4, Figure 2] The central 'no overfitting' claim is not verifiable as reported. The Hit@3 comparison between BCE loss and contrastive loss does not state whether the metric is computed on a held-out split, how the split is constructed relative to the 10x pretraining data, or whether the same evaluation protocol is used for both losses. Without this information, the flat contrastive curve could reflect memorization of the training set rather than generalization. The caption's assertion that contrastive loss 'generalizes well over epochs' requires a held-out evaluation, and the paper should specify the split, the overlap with training data, and the evaluation protocol.
[§3, §4, Table 1] The causal attribution to contrastive loss is underdetermined by the existing ablations. The paper states Stage 1 uses '10x more engagement data from various sources compared to downstream models,' but no experiment holds the pretraining data and epoch count fixed while varying only the loss. Table 1 compares single-stage versus two-stage training, but all arms share the same contrastive pretraining loss and data; Figure 2 compares BCE versus contrastive only in the pretraining stage and does not feed either set of embeddings into the downstream model. Thus the +2.2% online gain (Table 2) could be due to larger data coverage, multi-epoch pretraining, or the two-stage separation itself, rather than to the contrastive loss. An ablation with identical pretraining data and epochs, varying only the loss, plus a downstream comparison of BCE-pretrained versus contrastive-pretrained embeddings,
[§3, §1 and Figure 2 caption] The mechanistic claim that contrastive loss 'mitigates the one-epoch overfitting phenomenon by reducing the effective dimensionality of tail entries' is asserted without any formal definition or measurement. 'Effective dimensionality' is not quantified (e.g., via eigenvalue spectra, participation ratios, or norm analysis), and no empirical evidence links the observed Hit@3 curves to this mechanism. As stated, this is a post hoc interpretation. It should either be supported with measurements or explicitly labeled as a hypothesis that the experiments do not directly test.
[§4, Table 2] The online A/B experiment reporting is too sparse to assess the significance of the 2.2% site-wide claim. The paper states only that the experiments ran for two weeks and reports engagement lifts of +1.11% and +1.09% for Homefeed and Related Pins. It does not report the number of users, the confidence intervals, the computation of the aggregate 2.2%, or whether the two surfacing tests overlapped. Given that Table 1 includes uncertainty intervals for offline Hit@3, the absence of any uncertainty or statistical testing for the online metrics is a gap in the central empirical claim.

minor comments (6)

[Figure 2 caption] 'Binary entropy loss' should be 'binary cross-entropy loss' (BCE).
[Figure 1] The text in the figure contains 'Surface' with inconsistent capitalization and an apparent missing label ('Surface 1 Surface'). Minor readability issue.
[Section 2, 'Trade-off between Model Complexity and Training Cost'] The phrase 'often require extended training windows to cover tail IDs and reach convergence' would benefit from a precise definition of 'training windows' and how it relates to the storage cost mentioned below.
[Overall] The paper uses 'we' inconsistently ('we showed', 'we propose', etc.) in Section 4 and elsewhere; a light copyedit would improve readability.
[Conclusion] The conclusion repeats the abstract without adding interpretation or limitations. One or two sentences on the main remaining threats to validity (e.g., the absence of loss-isolation ablation) would make the paper more self-aware.
[References] Some references are to arXiv preprints rather than published versions; if the published versions are available, citing them would help readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is supported by offline ablations and live A/B measurements, not by a definitional reduction or a load-bearing self-citation chain.

full rationale

The paper's main claim is empirical: a two-stage training scheme where ID embeddings are pre-trained with a contrastive loss and then fine-tuned in downstream models yields better generalization and a site-wide engagement gain. The contrastive loss in Eq. (1) is borrowed from prior work [8], which is an independent source and is not used to justify the present paper's own results. The offline ablation in Table 1 compares single-stage, two-stage frozen, and two-stage fine-tuned training; the online A/B test in Table 2 measures engagement against production. None of these outcomes is a fitted parameter renamed as a prediction, and no equation defines the predicted metric in terms of the model's inputs by construction. The paper's informal explanation that contrastive loss reduces the 'effective dimensionality of tail entries' is post hoc and underspecified, but it is not itself the evidence for the result—the Hit@3 and engagement measurements are. The underspecified Hit@3 evaluation protocol (e.g., whether the metric is held-out) is a validity concern, not circularity: it could undermine the overfitting claim but does not show that the claim is equivalent to its inputs. There are no load-bearing self-citations by the current authors; the acknowledged contributions of colleagues are not used as evidence. Therefore the derivation chain is self-contained as an empirical engineering result, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The paper's central results are empirical and depend on standard contrastive learning plus several unstated engineering choices: exact number of negatives, epoch count, data splits, and the transferability of minimal-model embeddings to complex downstream models. The 'effective dimensionality' construct is an invented explanatory quantity without independent evidence.

free parameters (3)

Learned temperature tau = not reported
Appears in the contrastive loss in Eq. (1); the training behavior depends on it but no value or schedule is given.
Number of negatives N = not reported
Combination of in-batch and uniform random negatives in Eq. (1); chosen by hand and not reported.
Number of pre-training epochs = not reported
The paper says 'multi-epoch' pre-training, but the exact count is unspecified, and it is load-bearing for the overfitting claim in Figure 2.

assumptions (4)

domain assumption One-epoch overfitting is caused by power-law ID distributions in which tail IDs have more dimensions of freedom than training samples.
Stated in the Introduction as the cause of the one-epoch phenomenon; no derivation or direct measurement is provided.
domain assumption Contrastive loss with in-batch and random negatives reduces the effective dimensionality of tail IDs.
Asserted in Section 3 as the mechanism behind the observed generalization; no quantitative definition or estimate of effective dimensionality is given.
domain assumption Minimal dot-product pretraining on aggregated engagement data transfers to complex downstream ranking models.
The two-stage approach presupposes this transfer; Table 1 provides empirical support but no analysis of distribution mismatch or task alignment.
domain assumption Hit@3 in Figure 2 and Table 1 is evaluated on held-out interactions.
The evaluation protocol is not stated in Section 4; if the metric is computed on training data, the no-overfitting conclusion would not follow.

invented entities (1)

Effective dimensionality of tail entries
purpose: Explains why contrastive loss avoids one-epoch overfitting in Section 3.
Introduced as the mechanism for the observed effect, but never defined, measured, or given a falsifiable handle outside this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training." pith.science (2026). https://pith.science/paper/5FPCWWTE

@misc{pith2026250818700,
  author       = {Pith},
  title        = {Pith review of: Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FPCWWTE}},
  note         = {Machine review of arXiv:2508.18700}
}

read the original abstract

ID-based embeddings are widely used in web-scale online recommendation systems. However, their susceptibility to overfitting, particularly due to the long-tail nature of data distributions, often limits training to a single epoch, a phenomenon known as the "one-epoch problem." This challenge has driven research efforts to optimize performance within the first epoch by enhancing convergence speed or feature sparsity. In this study, we introduce a novel two-stage training strategy that incorporates a pre-training phase using a minimal model with contrastive loss, enabling broader data coverage for the embedding system. Our offline experiments demonstrate that multi-epoch training during the pre-training phase does not lead to overfitting, and the resulting embeddings improve online generalization when fine-tuned for more complex downstream recommendation tasks. We deployed the proposed system in live traffic at Pinterest, achieving significant site-wide engagement gains.

Figures

Figures reproduced from arXiv: 2508.18700 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

[8]

Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- former: Sequence modeling for user representation at pinterest. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining . 3702–3712

work page 2022
[1]

On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models

Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop, Kevin Regan, Gil I. Shamir, Rakesh Shivanna, and Qiqi Yan. 2022. On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models. arXiv:2209.05310 [cs.IR]

work page Pith review arXiv 2022
[2]

Qiwei Chen, Yue Xu, Changhua Pei, Shanshan Lv, Tao Zhuang, and Junfeng Ge. 2022. Efficient Long Sequential User Data Modeling for Click-Through Rate Prediction. arXiv preprint arXiv:2209.12212 (2022)

arXiv 2022
[3]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198

2016
[4]

Michalis Faloutsos, Petros Faloutsos, and Christos Faloutsos. 1999. On power-law relationships of the internet topology. ACM SIGCOMM computer communication review 29, 4 (1999), 251–262

work page 1999
[5]

Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. 2005. Graphs over time: densification laws, shrinking diameters and possible explanations. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining. 177–187

work page 2005
[6]

Zhuoran Liu, Leqi Zou, Xuan Zou, Caihua Wang, Biao Zhang, Da Tang, Bolin Zhu, Yijie Zhu, Peng Wu, Ke Wang, et al. 2022. Monolith: real time recommendation system with collisionless embedding table.arXiv preprint arXiv:2209.07663 (2022)

arXiv 2022
[7]

Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole- Jean Wu, Alisson G Azzolini, et al. 2019. Deep learning recommendation model for personalization and recommendation systems.arXiv preprint arXiv:1906.00091 (2019)

arXiv 2019

Show all 15 references

[9]

Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Practice on long sequential user behavior modeling for click-through rate prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 2671–2679

2019
[10]

Anima Singh, Trung Vu, Raghunandan Keshavan, Nikhil Mehta, Xinyang Yi, Lichan Hong, Lukasz Heldt, Li Wei, Ed Chi, and Maheswaran Sathiamoorthy
[11]

Chunxing Yin, Bilge Acun, Carole-Jean Wu, and Xing Liu. 2021. Tt-rec: Tensor train compression for deep learning recommendation models. Proceedings of Machine Learning and Systems 3 (2021), 448–462

2021
[12]

Zhao-Yu Zhang, Xiang-Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng, and Bo Zheng. 2022. Towards understanding the overfitting phenomenon of deep click-through rate models. In Proceedings of the 31st ACM international conference on information & knowledge mana...

2022
[13]

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...

2018
[14]

Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. 2021. Open benchmarking for click-through rate prediction. In Proceedings of the 30th ACM international conference on information & knowledge management . 2759–2769

2021
[2023]

arXiv preprint arXiv:2306.08121 (2023)

Better Generalization with Semantic IDs: A case study in Ranking for Recommendations. arXiv preprint arXiv:2306.08121 (2023)

2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [8]

Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- former: Sequence modeling for user representation at pinterest. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining . 3702–3712

work page 2022

[2] [1]

On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models

Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop, Kevin Regan, Gil I. Shamir, Rakesh Shivanna, and Qiqi Yan. 2022. On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models. arXiv:2209.05310 [cs.IR]

work page Pith review arXiv 2022

[3] [2]

Qiwei Chen, Yue Xu, Changhua Pei, Shanshan Lv, Tao Zhuang, and Junfeng Ge. 2022. Efficient Long Sequential User Data Modeling for Click-Through Rate Prediction. arXiv preprint arXiv:2209.12212 (2022)

arXiv 2022

[4] [3]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198

2016

[5] [4]

Michalis Faloutsos, Petros Faloutsos, and Christos Faloutsos. 1999. On power-law relationships of the internet topology. ACM SIGCOMM computer communication review 29, 4 (1999), 251–262

work page 1999

[6] [5]

Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. 2005. Graphs over time: densification laws, shrinking diameters and possible explanations. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining. 177–187

work page 2005

[7] [6]

Zhuoran Liu, Leqi Zou, Xuan Zou, Caihua Wang, Biao Zhang, Da Tang, Bolin Zhu, Yijie Zhu, Peng Wu, Ke Wang, et al. 2022. Monolith: real time recommendation system with collisionless embedding table.arXiv preprint arXiv:2209.07663 (2022)

arXiv 2022

[8] [7]

Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole- Jean Wu, Alisson G Azzolini, et al. 2019. Deep learning recommendation model for personalization and recommendation systems.arXiv preprint arXiv:1906.00091 (2019)

arXiv 2019