{"id":"d741ac56-3a8f-4dff-b5d8-f499a08ebcbe","arxiv_id":"2509.04337","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-computed user and Pin embeddings from multi-tower models improve Pinterest ad ranking by small but statistically significant margins.","lead":"A Pinterest team describes a two-stage system that learns user and Pin embeddings offline, then feeds them into ad ranking models. The paper reports small but consistent offline and online gains, including a 1.4-2.8% click-through lift in A/B tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Back-inference leakage likely inflates offline AUC lifts, and the online A/B tests a configuration selected under the same leaked metric.","rationale":"The reader's weakest assumption correctly identifies the back-inference leakage. I agree this is the most load-bearing concern because it undermines both the offline evidence and the model selection process. The online A/B provides some support, but it is a single test of a configuration selected under a leaky offline metric. The condition of the reader's verdict—remove or justify the back-inference step—is necessary to validate the central claim. Therefore my analysis does not change the CONDITIONAL verdict.","tokens_in":8435,"tokens_out":4117,"duration_ms":36276,"concrete_test":"Re-run the offline experiments for the downstream CTR and CVR tasks using a clean temporal split: train the upstream model on data through day T, freeze it, and generate embeddings for a held-out evaluation week (days T+1 to T+7) using only forward inference (no back inference). Train the downstream models on days T+1 to T+7 with these embeddings and compare their ROC-AUC lifts against the respective controls to those in Tables 2 and 4. If the lifts shrink to noise levels (e.g., below the reported 0.05% significance threshold), the back-inference step is the source of the inflated offline gains. Also, repeat the w selection (Table 1) with leak-free embeddings; if a different w is optimal, the deployed configuration was chosen on a contaminated metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the back-inference step in Appendix B.3. The authors generate entity embeddings by running the latest upstream model snapshot over the same training window used to train downstream CTR/CVR models. Because the upstream models are trained on click/conversion labels, these embeddings can encode label information for the very instances the downstream model sees. The authors acknowledge this 'may appear to introduce leakage' but retain it because it 'enriches' representations. This is not a benign enrichment: if leakage occurs, the offline ROC-AUC lifts in Tables 2 and 4 (up to 0.19% and 0.22%) are not evidence of genuine representation quality, and the offline tuning that selects w=0.8 and the four-embedding combination is fitted to a contaminated metric. The online A/B results are the only clean evidence, but they are a single evaluation of a configuration already selected under the leaked metric, so their confirmatory value is reduced. Without a leak-free offline evaluation, the paper's claim of 'notable performance improvements in both offline and online settings' is not fully supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a decoupled upstream-downstream framework (DERM) for learning user and Pin embeddings at Pinterest. Upstream multi-tower DHEN models are trained on ad CTR and CVR datasets with supervised tasks plus a sampled-softmax contrastive loss; the resulting entity embeddings are refreshed daily, aggregated with a moving average, and then fed into downstream CTR and CVR ranking models. Offline experiments report ROC-AUC lifts up to 0.19% on CTR and 0.22% on CVR, while online A/B tests report a platform-wise CTR lift of 1.38%, gCTR lift of 1.96%, and a 1.61% CPA reduction. The central claim is that the framework yields notable performance improvements in both offline and online settings.","tokens_in":8699,"tokens_out":4658,"duration_ms":42933,"significance":"If the offline evidence were clean, the paper would offer a useful industrial template for reusable entity representations with cross-domain transfer, and the ablation across embedding types plus the cost analysis would be valuable. The framework is plausible and has been deployed in production, which is a concrete strength. However, the offline evaluation is compromised by a back-inference procedure that can leak click/conversion labels into the embeddings, and the online A/B test is the only trustworthy evidence. The paper also ships no code or data, so the reported gains cannot be independently reproduced; nevertheless, the system-level description and the explicit acknowledgment of the leakage risk are to the authors' credit.","major_comments":[{"comment":"The manuscript explicitly says that entity embeddings are produced by 'back inference on the days within the batch training window using the latest model snapshot' and acknowledges that this 'may appear to introduce leakage,' but retains it because it 'enriches' representations. Since the upstream models are trained on click/conversion labels, this procedure can encode label information for the exact instances used to train and evaluate downstream CTR/CVR models. All offline lifts in Tables 2 and 4, and the heuristic comparison in Table 1, are therefore not clean evidence of representation quality. This is load-bearing: the abstract claims 'notable performance improvements in both offline and online settings,' and the offline results are used to select w=0.8 and the four-embedding combination that is later A/B tested. Please provide a leak-free offline evaluation—for example, embeddings","section":"Appendix B.3; Tables 2 and 4"},{"comment":"The offline gains are extremely small (0.02%–0.22% ROC-AUC) and are reported without error bars, confidence intervals, or significance tests; the footnote in Table 4 states a 0.05% ROC-AUC lift as a significance threshold but provides no variance estimate. Similarly, Table 3 states 'all stats-sig' without p-values, confidence intervals, sample sizes, experiment duration, or correction for multiple comparisons. Given the leakage concern in Appendix B.3, the online A/B result carries the full weight of the paper's central claim, and its current reporting is insufficient to judge whether the observed lifts are reproducible effects rather than selected-maximum artifacts. Please report standard errors or confidence intervals and provide full details of the A/B design.","section":"Section 4, Tables 3 and 4"}],"minor_comments":[{"comment":"The moving-average equation E_agg(t) = w E_agg(t-1) + (1-w) E_daily(t) is unnumbered; please number it to allow precise reference.","section":"Section 3.1"},{"comment":"The sentence 'Similar trends can be observed with other DERM embeddings in different downstream tasks' is unsupported by data. Either include the results or remove the claim.","section":"Table 1"},{"comment":"The statement that cosine similarity analysis 'demonstrates minimal daily variations' should report the actual statistics (mean, quantiles) rather than an informal assertion.","section":"Appendix C.1"},{"comment":"The abstract and conclusion describe the improvements as 'notable,' but the offline AUC lifts are at most 0.22%. Please qualify the claim or provide a contextual baseline so readers can calibrate the magnitude.","section":"Abstract and Conclusion"},{"comment":"The term 'leakage' is used in quotes in B.3; if the procedure is retained, the paper should at least discuss why the leakage direction is benign for the downstream task and how the embedding generation time is aligned with the downstream label time. As written, the retention is ad hoc.","section":"Appendix B.3 and Appendix C.2"},{"comment":"The manuscript contains a 'SPEAKER BIO' section and ACM 'Conference'17' template placeholders (dates, conference name). These should be removed or updated before submission.","section":"Back matter"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid industry-systems contribution, but the leakage acknowledgment in Appendix B.3 is likely to be the central concern for any careful reviewer. The authors should be asked to redo the offline evaluation with strictly temporal splits and to report uncertainty/experimental details for the online test. If they cannot provide a leak-free offline evaluation, the paper's scientific claims would rest solely on a single online experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real production paper, not a research breakthrough. The new part is the specific combination — multi-tower DHEN entity towers, a contrastive sampled-softmax loss between user and Pin towers, a projection layer for cost, and moving-average embedding refresh — applied to Pinterest's ads ranking stack. Each component is known and mostly cited; the authors themselves build directly on their earlier DHEN practice paper and on Meta's scaling user modeling. What is new is the system-level result and the honest writeup of engineering choices.\n\nWhat the paper does well: it gives enough architectural detail to be reproducible inside a similar industrial setting, it reports online A/B results with actual traffic splits, and it is unusually transparent in Appendix B.3. It tells you that embeddings are generated from the latest model snapshot over the same training window used to train downstream models, and that this 'may appear to introduce leakage' but is retained because it 'enriches' representations. That is not a benign enrichment. If the upstream models are trained on click/conversion labels, then re-inferring embeddings over the same window can encode label information into the features of the very instances the downstream model sees. That likely inflates the offline ROC-AUC lifts in Tables 2 and 4 (up to 0.19% and 0.22%). The authors might be right that in practice it works, but the offline numbers are not clean evidence.\n\nThere is also a mild circularity: the moving-average weight w=0.8 is selected by testing on the downstream CVR task, and that same weight is then reported as the best choice. No error bars or confidence intervals for the offline lifts, and the online lifts are marked 'stats-sig' without p-values. All of this is typical for industrial papers, and none of it is fatal, but it means the only unambiguously clean evidence is the online A/B test. And that test evaluates a configuration already picked under the contaminated offline metric, so its confirmatory value is a bit lower.\n\nWho is this for: people working on large-scale ad ranking or embedding-based transfer learning at a platform company. This is a useful data point about what works in production, and a good case study in evaluation leakage for the rest of us.\n\nMy summary: the paper deserves a serious referee. The architecture is legitimate, the deployment is real, and the online results are plausible. But the authors should either drop the back-inference from the offline evaluation or show it doesn't affect the reported lifts, and they should give some statistical detail for the online tests. Send it to a venue that handles applied recommender systems, with a request for revisions.","headline":"Honest production paper: real deployment, known components, but back-inference muddies the offline evidence; online A/B is the clean part.","tokens_in":9183,"tokens_out":3089,"would_cite":true,"duration_ms":24249,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three-tower embeddings lift Pinterest ads CTR by 1.38% online.","keywords":["entity representation learning","embeddings","ads ranking","CTR prediction","CVR prediction","multi-task learning","contrastive learning","upstream-downstream framework"],"falsifier":"Recompute the offline AUC lifts using strictly causal embedding generation—each day's embeddings produced only by the model snapshot available that day—and compare with the reported back-inferred numbers. If the causal lifts drop to zero while back-inferred lifts stay positive, the offline gains are leakage, leaving the online A/B test as the only reliable evidence.","tokens_in":8367,"feed_emoji":"📈","tokens_out":4219,"duration_ms":38215,"temperature":0.7,"pith_summary":"Ads rankings suffer because each surface and ad type builds its own embeddings, so users and items get fragmented representations. This paper argues that a decoupled upstream model—trained on CTR and CVR data with three towers, a contrastive self-supervised loss, and a daily moving-average refresh—can produce user and Pin embeddings that transfer across downstream retrieval and ranking tasks. The authors report that feeding these precomputed embeddings into existing CTR and CVR models improved offline ROC-AUC by up to 0.19% and 0.22%, and online CTR by 1.38%, gCTR by 1.96%, and reduced CPA by 1.61% in a deployed A/B test. If the framework works as claimed, a shared representation layer can replace per-task embedding tables without retraining the downstream stack.","feed_headline":"Pinterest ads CTR up 1.38% with shared user-Pin embeddings","feed_subtitle":"A decoupled three-tower model builds refreshable embeddings that lift CTR and conversion across ads ranking tasks.","key_machinery":"The central object is the DERM embedding pair (user vector, Pin vector) produced by separate entity towers, kept free of context contamination by an overtower, and stabilized by moving-average aggregation E_agg(t) = 0.8*E_agg(t-1) + 0.2*E_daily(t). The contrastive sampled-softmax loss with trainable temperature and frequency-bias correction is what pulls user and Pin embeddings into a shared space. These vectors enter downstream CTR/CVR models as concatenated features, with a projection layer to keep DCNv2 cost manageable.","core_discovery":"The paper's central claim is that entity representations for ads ranking can be learned once, upstream, from both ad-engagement and ad-conversion data, and then served as fixed input features to many downstream models. The proposed Decoupled Entity Representation Model (DERM) uses a User tower and a Pin tower over DHEN backbones, plus an Overall Interaction Tower that absorbs contextual and interaction features so the pure entity embeddings stay stable. A sampled softmax contrastive loss aligns positive user-Pin pairs against in-batch negatives with frequency-bias correction, and a moving average with weight 0.8 on the previous day's embedding smooths the daily refresh. In offline tests, all","pith_inferences":["The reported online lifts come from a single deployment; the framework's generality beyond Pinterest's surfaces is untested but plausible for any two-sided marketplace with separate retrieval and ranking stages.","Back-inferring embeddings with the latest model snapshot over historical data may leak future information; if so, the offline gains are optimistic and the online A/B test is the only clean estimate.","The moving-average weight of 0.8 implies entity semantics drift slowly; one could test whether per-entity or per-domain weights improve stability further.","A unified upstream model producing CTR and CVR embeddings in one shared space could reduce training cost while preserving cross-domain transfer, which the paper lists as future work."],"forward_implications":["Ad retrieval and ranking can share one embedding layer instead of per-surface embedding tables.","Daily refresh with weighted moving average keeps coverage and stability without real-time inference.","The contrastive objective transfers knowledge across CTR and CVR domains, as evidenced by the CVR task gaining from CTR-trained embeddings.","A projection layer reduces DCNv2 cost and saves about $212K per year in inference infrastructure with only a 0.01% AUC drop.","The same upstream-downstream pattern could be reused for other entity types, such as queries and advertisers."],"supporting_citations":[{"why":"Classic two-tower model that trains on dot-product similarity; DERM contrasts its overtower design against this baseline.","marker":"[5]"},{"why":"Approximate nearest neighbor search; motivates why embeddings are useful for candidate retrieval in the ranking pipeline.","marker":"[7]"},{"why":"PinnerFormer sequence modeling for user representation; supplies prior self-supervised representation learning inspiration.","marker":"[10]"},{"why":"CLIP-style contrastive learning; source of the sampled softmax loss with in-batch negatives and trainable temperature.","marker":"[12]"},{"why":"DCNv2 feature interaction layer used in downstream CTR model; its quadratic complexity motivates the projection layer.","marker":"[14]"},{"why":"DHEN backbone; supplies the multi-layer deep hierarchical ensemble network used in the entity towers.","marker":"[17]"},{"why":"Previous large-scale online user representation work at Meta; the paper positions DERM as extending it from users only to users and Pins with contrastive learning.","marker":"[18]"},{"why":"Pinterest practice for deep hierarchical ensemble networks in CVR prediction; the upstream approach builds directly on it and the CVR downstream model shares its architecture.","marker":"[19]"}],"fun_headline_variants":["Decoupled embeddings lift Pinterest ad CTR and CVR","Shared user-Pin embeddings boost Pinterest ads ranking","Upstream-downstream model improves Pinterest ad metrics","Bi-weekly refreshed embeddings enhance ad ranking at Pinterest","Entity embeddings from diverse signals power Pinterest ads"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The offline evaluation assumes that back-inferring embeddings with the latest model snapshot over historical training data does not leak future label information into the downstream models; the paper admits this 'may appear to introduce leakage' but keeps it because it enriches representations.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled embeddings lift Pinterest ad CTR and CVR","Shared user-Pin embeddings boost Pinterest ads ranking","Upstream-downstream model improves Pinterest ad metrics","Bi-weekly refreshed embeddings enhance ad ranking at Pinterest","Entity embeddings from diverse signals power Pinterest ads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1321,"prompt_tokens":666,"completion_tokens":655,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":410,"completion_tokens_details":{"reasoning_tokens":583}},"tokens_in":410,"tokens_out":655,"duration_ms":6604,"temperature":1.0,"reasoning_tokens":583,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:12:10.700894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the offline AUC lifts using strictly causal embedding generation—each day's embeddings produced only by the model snapshot available that day—and compare with the reported back-inferred numbers. If the causal lifts drop to zero while back-inferred lifts stay positive, the offline gains are leakage, leaving the online A/B test as the only reliable evidence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PinnerFormer sequence modeling for user representation; supplies prior self-supervised representation learning inspiration."},{"cited_title":"On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction","cited_arxiv_id":"2504.08169","evidence_quote":"Pinterest practice for deep hierarchical ensemble networks in CVR prediction; the upstream approach builds directly on it and the CVR downstream model shares its architecture."}],"review_version":1}