Pith. sign in

REVIEW 2 major objections 6 minor 19 references

Three-tower embeddings lift Pinterest ads CTR by 1.38% online.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pre-computed user and Pin embeddings from multi-tower models improve Pinterest ad ranking by small but statistically significant margins.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Honest production paper: real deployment, known components, but back-inference muddies the offline evidence; online A/B is the clean part. the 2 major comments →

arxiv 2509.04337 v1 pith:LC3G5MGP submitted 2025-09-04 cs.IR cs.AIcs.LG

Decoupled Entity Representation Learning for Pinterest Ads Ranking

Jie Liu , Yinrui Li , Jiankai Sun , Kungang Li , Han Sun , Sihan Wang , Huasen Wu , Siyuan Gao
show 7 more authors
Paulo Soares Nan Li Zhifang Liu Haoyang Li Siping Ji Ling Leng Prathibha Deshikachar
This is my paper
classification cs.IR cs.AIcs.LG
keywords entity representation learningembeddingsads rankingCTR predictionCVR predictionmulti-task learningcontrastive learningupstream-downstream framework
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ads rankings suffer because each surface and ad type builds its own embeddings, so users and items get fragmented representations. This paper argues that a decoupled upstream model—trained on CTR and CVR data with three towers, a contrastive self-supervised loss, and a daily moving-average refresh—can produce user and Pin embeddings that transfer across downstream retrieval and ranking tasks. The authors report that feeding these precomputed embeddings into existing CTR and CVR models improved offline ROC-AUC by up to 0.19% and 0.22%, and online CTR by 1.38%, gCTR by 1.96%, and reduced CPA by 1.61% in a deployed A/B test. If the framework works as claimed, a shared representation layer can replace per-task embedding tables without retraining the downstream stack.

Core claim

The paper's central claim is that entity representations for ads ranking can be learned once, upstream, from both ad-engagement and ad-conversion data, and then served as fixed input features to many downstream models. The proposed Decoupled Entity Representation Model (DERM) uses a User tower and a Pin tower over DHEN backbones, plus an Overall Interaction Tower that absorbs contextual and interaction features so the pure entity embeddings stay stable. A sampled softmax contrastive loss aligns positive user-Pin pairs against in-batch negatives with frequency-bias correction, and a moving average with weight 0.8 on the previous day's embedding smooths the daily refresh. In offline tests, all

What carries the argument

The central object is the DERM embedding pair (user vector, Pin vector) produced by separate entity towers, kept free of context contamination by an overtower, and stabilized by moving-average aggregation E_agg(t) = 0.8*E_agg(t-1) + 0.2*E_daily(t). The contrastive sampled-softmax loss with trainable temperature and frequency-bias correction is what pulls user and Pin embeddings into a shared space. These vectors enter downstream CTR/CVR models as concatenated features, with a projection layer to keep DCNv2 cost manageable.

Load-bearing premise

The offline evaluation assumes that back-inferring embeddings with the latest model snapshot over historical training data does not leak future label information into the downstream models; the paper admits this 'may appear to introduce leakage' but keeps it because it enriches representations.

What would settle it

Recompute the offline AUC lifts using strictly causal embedding generation—each day's embeddings produced only by the model snapshot available that day—and compare with the reported back-inferred numbers. If the causal lifts drop to zero while back-inferred lifts stay positive, the offline gains are leakage, leaving the online A/B test as the only reliable evidence.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Ad retrieval and ranking can share one embedding layer instead of per-surface embedding tables.
  • Daily refresh with weighted moving average keeps coverage and stability without real-time inference.
  • The contrastive objective transfers knowledge across CTR and CVR domains, as evidenced by the CVR task gaining from CTR-trained embeddings.
  • A projection layer reduces DCNv2 cost and saves about $212K per year in inference infrastructure with only a 0.01% AUC drop.
  • The same upstream-downstream pattern could be reused for other entity types, such as queries and advertisers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported online lifts come from a single deployment; the framework's generality beyond Pinterest's surfaces is untested but plausible for any two-sided marketplace with separate retrieval and ranking stages.
  • Back-inferring embeddings with the latest model snapshot over historical data may leak future information; if so, the offline gains are optimistic and the online A/B test is the only clean estimate.
  • The moving-average weight of 0.8 implies entity semantics drift slowly; one could test whether per-entity or per-domain weights improve stability further.
  • A unified upstream model producing CTR and CVR embeddings in one shared space could reduce training cost while preserving cross-domain transfer, which the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes a decoupled upstream-downstream framework (DERM) for learning user and Pin embeddings at Pinterest. Upstream multi-tower DHEN models are trained on ad CTR and CVR datasets with supervised tasks plus a sampled-softmax contrastive loss; the resulting entity embeddings are refreshed daily, aggregated with a moving average, and then fed into downstream CTR and CVR ranking models. Offline experiments report ROC-AUC lifts up to 0.19% on CTR and 0.22% on CVR, while online A/B tests report a platform-wise CTR lift of 1.38%, gCTR lift of 1.96%, and a 1.61% CPA reduction. The central claim is that the framework yields notable performance improvements in both offline and online settings.

Significance. If the offline evidence were clean, the paper would offer a useful industrial template for reusable entity representations with cross-domain transfer, and the ablation across embedding types plus the cost analysis would be valuable. The framework is plausible and has been deployed in production, which is a concrete strength. However, the offline evaluation is compromised by a back-inference procedure that can leak click/conversion labels into the embeddings, and the online A/B test is the only trustworthy evidence. The paper also ships no code or data, so the reported gains cannot be independently reproduced; nevertheless, the system-level description and the explicit acknowledgment of the leakage risk are to the authors' credit.

major comments (2)
  1. [Appendix B.3; Tables 2 and 4] The manuscript explicitly says that entity embeddings are produced by 'back inference on the days within the batch training window using the latest model snapshot' and acknowledges that this 'may appear to introduce leakage,' but retains it because it 'enriches' representations. Since the upstream models are trained on click/conversion labels, this procedure can encode label information for the exact instances used to train and evaluate downstream CTR/CVR models. All offline lifts in Tables 2 and 4, and the heuristic comparison in Table 1, are therefore not clean evidence of representation quality. This is load-bearing: the abstract claims 'notable performance improvements in both offline and online settings,' and the offline results are used to select w=0.8 and the four-embedding combination that is later A/B tested. Please provide a leak-free offline evaluation—for example, embeddings
  2. [Section 4, Tables 3 and 4] The offline gains are extremely small (0.02%–0.22% ROC-AUC) and are reported without error bars, confidence intervals, or significance tests; the footnote in Table 4 states a 0.05% ROC-AUC lift as a significance threshold but provides no variance estimate. Similarly, Table 3 states 'all stats-sig' without p-values, confidence intervals, sample sizes, experiment duration, or correction for multiple comparisons. Given the leakage concern in Appendix B.3, the online A/B result carries the full weight of the paper's central claim, and its current reporting is insufficient to judge whether the observed lifts are reproducible effects rather than selected-maximum artifacts. Please report standard errors or confidence intervals and provide full details of the A/B design.
minor comments (6)
  1. [Section 3.1] The moving-average equation E_agg(t) = w E_agg(t-1) + (1-w) E_daily(t) is unnumbered; please number it to allow precise reference.
  2. [Table 1] The sentence 'Similar trends can be observed with other DERM embeddings in different downstream tasks' is unsupported by data. Either include the results or remove the claim.
  3. [Appendix C.1] The statement that cosine similarity analysis 'demonstrates minimal daily variations' should report the actual statistics (mean, quantiles) rather than an informal assertion.
  4. [Abstract and Conclusion] The abstract and conclusion describe the improvements as 'notable,' but the offline AUC lifts are at most 0.22%. Please qualify the claim or provide a contextual baseline so readers can calibrate the magnitude.
  5. [Appendix B.3 and Appendix C.2] The term 'leakage' is used in quotes in B.3; if the procedure is retained, the paper should at least discuss why the leakage direction is benign for the downstream task and how the embedding generation time is aligned with the downstream label time. As written, the retention is ad hoc.
  6. [Back matter] The manuscript contains a 'SPEAKER BIO' section and ACM 'Conference'17' template placeholders (dates, conference name). These should be removed or updated before submission.

Circularity Check

1 steps flagged

Offline evaluation is partially circular due to back-inference leakage; online A/B tests provide independent grounding.

specific steps
  1. fitted input called prediction [Appendix B.3 (Inference), reflected in Tables 2 and 4]
    "Each day, the model snapshot from the incremental training is used to generate entity embeddings from the same training data in an offline manner. To extend the availability of trained embeddings over more days, we can perform back inference on the days within the batch training window using the latest model snapshot. While this approach may appear to introduce ’leakage, ’ it actually enriches the entity representations and has proven effective in enhancing downstream model performance."

    The DERM embedding is a fitted quantity: the latest upstream snapshot is trained on click/conversion labels and then applied to the same days' data that feed downstream CTR/CVR training and offline evaluation. The downstream feature can therefore carry label information for the very samples whose AUC is being measured, so the offline lifts in Tables 2/4 are not clean evidence of representation quality. The paper's stated justification ('enriches the entity representations') is exactly the leakage channel, and retaining back-inference because it 'enhanc[es] downstream model performance' selects the feature by a contaminated metric. Online A/B results remain independent because they use freshly served embeddings.

full rationale

The core architecture and training procedure are self-contained: DERM's multi-tower design, contrastive loss, and downstream integration are evaluated by controlled comparisons, and no central claim depends on a self-citation chain or an imported uniqueness theorem. The moving-average weight w=0.8 is a hyperparameter tuned on the CVR task (Table 1); while this is model selection on the evaluation metric, it is not circular. The trainable softmax temperature is a standard parameter. The one substantive circular dependency is the back-inference step in Appendix B.3: embeddings are generated from the same labeled training window used to train and score downstream models, which the paper itself admits 'may appear to introduce leakage.' This makes the offline AUC lifts partially circular, but the online A/B tests provide independent grounding, so the overall circularity is moderate (3/10).

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new entities are invented; the framework recombines known components. The main fitted quantities are the aggregation weight, the softmax temperature, and the projection dimension.

free parameters (3)
  • moving average weight w = 0.8
    Tuned on CVR downstream task offline (Table 1) and then applied across tasks.
  • softmax temperature tau = trainable
    Trainable temperature in sampled softmax loss (Section 3.1).
  • projection layer dimension = 512
    Chosen as best tradeoff between performance and cost; 512 gives 0.18% vs 0.19% ROC-AUC lift (Appendix D.1).
axioms (3)
  • domain assumption Production baseline models are appropriate controls
    All lifts are measured relative to Pinterest's production models, whose configurations are not described.
  • standard math In-batch sampled softmax with correction term Q estimates the true softmax
    The sampled softmax loss assumes the debiasing correction is valid; standard in the literature.
  • domain assumption AUC lift significance threshold of 0.05% is meaningful
    Appendix E.1 states 0.05% ROC-AUC lift is considered significant in Pinterest; no statistical test is provided.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoupled Entity Representation Learning for Pinterest Ads Ranking." pith.science (2026). https://pith.science/paper/LC3G5MGP

@misc{pith2026250904337,
  author       = {Pith},
  title        = {Pith review of: Decoupled Entity Representation Learning for Pinterest Ads Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LC3G5MGP}},
  note         = {Machine review of arXiv:2509.04337}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In this paper, we introduce a novel framework following an upstream-downstream paradigm to construct user and item (Pin) embeddings from diverse data sources, which are essential for Pinterest to deliver personalized Pins and ads effectively. Our upstream models are trained on extensive data sources featuring varied signals, utilizing complex architectures to capture intricate relationships between users and Pins on Pinterest. To ensure scalability of the upstream models, entity embeddings are learned, and regularly refreshed, rather than real-time computation, allowing for asynchronous interaction between the upstream and downstream models. These embeddings are then integrated as input features in numerous downstream tasks, including ad retrieval and ranking models for CTR and CVR predictions. We demonstrate that our framework achieves notable performance improvements in both offline and online settings across various downstream tasks. This framework has been deployed in Pinterest's production ad ranking systems, resulting in significant gains in online metrics.

Figures

Figures reproduced from arXiv: 2509.04337 by Han Sun, Haoyang Li, Huasen Wu, Jiankai Sun, Jie Liu, Kungang Li, Ling Leng, Nan Li, Paulo Soares, Prathibha Deshikachar, Sihan Wang, Siping Ji, Siyuan Gao, Yinrui Li, Zhifang Liu.

Figure 1
Figure 1. Figure 1: (top) DERM model architecture and (bottom) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 5 canonical work pages · 2 internal anchors

  1. [2]

    Paul Baltescu, Haoyu Chen, Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterest. arXiv:2205.11728 [cs.IR] https://arxiv.org/abs/ 2205.11728

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  2. [3]

    Ahmed El-Kishky, Thomas Markovich, Serim Park, Chetan Verma, Baekjin Kim, Ramy Eskander, Yury Malkov, Frank Portman, Sofía Samaniego, Ying Xiao, and Aria Haghighi. 2022. TwHIN: Embedding the Twitter Heterogeneous Information Network for Personalized Recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KD...

    arXiv 2022

  3. [4]

    Hamilton, Rex Ying, and Jure Leskovec

    William L. Hamilton, Rex Ying, and Jure Leskovec. 2018. Inductive Representation Learning on Large Graphs. arXiv:1706.02216 [cs.SI] https://arxiv.org/abs/1706. 02216

    Pith/arXiv arXiv 2018

  4. [5]

    Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding- based Retrieval in Facebook Search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’20) . ACM. doi:10.1145/3394486.3403305

    arXiv 2020

  5. [6]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton

  6. [7]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with GPUs. arXiv:1702.08734 [cs.CV] https://arxiv.org/abs/1702.08734

    Pith/arXiv arXiv 2017

  7. [8]

    Jordan and R.A

    M.I. Jordan and R.A. Jacobs. 1993. Hierarchical mixtures of experts and the EM algorithm. In Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan) , Vol. 2. 1339–1344 vol.2. doi:10.1109/IJCNN.1993. 716791

    work page doi:10.1109/ijcnn.1993 1993

  8. [9]

    Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, and Jure Leskovec. 2020. PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’20) . ACM. doi:10.1145/3394486.3403280

    work page arXiv 2020

  9. [10]

    Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- Former: Sequence Modeling for User Representation at Pinterest. In KDD. ACM, 3702–3712

    work page 2022

  10. [11]

    Andrew Qiu, Shubham Barhate, Hin Wai Lui, Runze Su, Rafael Rios Müller, Kungang Li, Ling Leng, Han Sun, Shayan Ehsani, and Zhifang Liu. 2025. The Evo- lution of Embedding Table Optimization and Multi-Epoch Training in Pinterest Decoupled Entity Representation Learning for Pinterest Ads Ranking Conference’17, July 2017, Washington, DC, USA Ads Conversion. ...

    Pith/arXiv arXiv 2025

  11. [12]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs.CV] https://arxiv.org/ abs/2103.00020

    Pith/arXiv arXiv 2021

  12. [13]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  13. [14]

    Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 1785–1797. doi:10.1145/3442381.3450078

    arXiv 2021

  14. [15]

    Zhiqiang Wang, Qingyun She, and Junlin Zhang. 2021. MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask. arXiv:2102.07619 [cs.IR] https://arxiv.org/abs/2102.07619

    Pith/arXiv arXiv 2021

  15. [16]

    Hamilton, and Jure Leskovec

    Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18) . ACM, 974–983. doi:10.1145/3219819.3219890

    arXiv 2018

  16. [17]

    Buyun Zhang, Liang Luo, Xi Liu, Jay Li, Zeliang Chen, Weilin Zhang, Xiaohan Wei, Yuchen Hao, Michael Tsang, Wenjun Wang, Yang Liu, Huayu Li, Yasmine Badr, Jongsoo Park, Jiyan Yang, Dheevatsa Mudigere, and Ellie Wen. 2022. DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction. arXiv:2203.11014 [cs.IR] https://arxiv.or...

    Pith/arXiv arXiv 2022

  17. [18]

    Wei Zhang, Dai Li, Chen Liang, Fang Zhou, Zhongke Zhang, Xuewei Wang, Ru Li, Yi Zhou, Yaning Huang, Dong Liang, Kai Wang, Zhangyuan Wang, Zhengxing Chen, Fenggang Wu, Minghai Chen, Huayu Li, Yunnan Wu, Zhan Shu, Mindi Yuan, and Sri Reddy. 2024. Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta. InCompanion Proc...

    arXiv 2024

  18. [19]

    Jinfeng Zhuang, Yinrui Li, Runze Su, Ke Xu, Zhixuan Shao, Kungang Li, Ling Leng, Han Sun, Meng Qi, Yixiong Meng, Yang Tang, Zhifang Liu, Qifei Shen, and Aayush Mudgal. 2025. On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction. arXiv:2504.08169 [cs.LG] https: //arxiv.org/abs/2504.08169 Conference’17, July 2017, Washingto...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [1991]

    Neural Computation 3, 1 (1991), 79–87

    Adaptive Mixtures of Local Experts. Neural Computation 3, 1 (1991), 79–87. doi:10.1162/neco.1991.3.1.79

    doi:10.1162/neco.1991.3.1.79 1991

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.