{"id":"2d2c2a0c-899f-424f-96a8-be4088b691ac","arxiv_id":"2507.12704","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A single pretrained model over user activity sequences improves save rates in Pinterest's Home Feed and Related Items ranking when fine-tuned per application, while deduplication and quantization keep serving costs neutral.","lead":"Pinterest pretrained a 20-billion-parameter model on two years of user activity data and fine-tuned it into its Home Feed and Related Items ranking systems, reporting gains in saves and engagement with fresh items. The paper also describes engineering tricks, such as deduplicated cross-attention, that let such a large model be served with neutral cost and latency.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20% cold-start claim in the abstract is not supported by Table 2: the 'boost' is the relative HIT@3 gap between two PinFM variants, not a measured engagement lift, and online Fresh Saves is +5.70%.","rationale":"The reader's conditional verdict is appropriate. My stress-test sharpens the same underlying weakness: the offline evidence for the cold-start contribution rests on a custom HIT@3 metric, an underspecified baseline, and a relative-lift presentation that makes the headline '20% increase in engagement' look larger than the online Fresh Saves result. I do not believe this warrants rejection, because the paper does provide online A/B evidence of engagement gains and deployment details, and the cold-start techniques may be genuinely effective. However, the abstract's specific quantitative claim should be reconciled with Table 2 and Table 7, and the authors should disclose raw metric values, confidence intervals, and baseline training effort. If those transparency conditions are met, the central claim is plausible; if they are not, the strongest advertised contribution is not verifiable from the paper.","tokens_in":16226,"tokens_out":11503,"duration_ms":135468,"concrete_test":"Ask the authors to report, for each row of Table 2, the raw HIT@3 rates with group counts and bootstrapped 95% confidence intervals, plus the training budget and hyperparameters used for the 'w/o PinFM' baseline. Then recompute the cold-start improvement as an absolute per-impression Fresh Save rate (e.g., Saves per thousand impressions on items younger than 7 days) for the deployed PinFM configuration versus the production baseline. If the absolute difference is not statistically significant or is materially below 20%, the abstract's cold-start claim must be revised; if the absolute difference survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim — that PinFM 'leads to a 20% increase in engagement with new items' — is not supported by the evidence as reported. Table 2 reports relative HIT@3 lifts against an unspecified 'w/o PinFM' baseline. The 'more than 20% boost' in the text is the arithmetic gap between PinFM-cs-none (HF 28d: -4.40%) and PinFM-cs-CIR-IDD-GSLT (+17.72%) on that custom proxy metric; it is the difference between two PinFM variants, not a measured 20% increase in user engagement. In the online A/B test, the deployed HF Fresh Saves lift is +5.70%, not 20%. The cold-start contribution is further weakened by the absence of confidence intervals or raw HIT@3 rates for Table 2, by the lack of any statement that the 'w/o PinFM' baseline received the same training budget and hyperparameter tuning, and by the paper's own admission that the cold-start fixes were introduced only after observing a negative fresh-item effect. The claim may be salvageable, but as stated it overstates what the reported data demonstrate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PinFM, a pretrained transformer-based foundation model over user activity sequences at Pinterest. The model is pretrained with contrastive next-token, multi-token, and future-token losses on two years of user actions, then fine-tuned into the Home Feed and Related Items ranking models. The authors introduce the Deduplicated Cross-Attention Transformer (DCAT) to share user-sequence computation across candidates, int4 post-training quantization for the billion-scale embedding table, and several cold-start mechanisms (candidate ID randomization, item-age dependent dropout, GraphSAGE embeddings, and a learnable token). Offline experiments report relative HIT@3 lifts across input-sequence variants, pretraining/fine-tuning losses, positive-action sets, vocabulary sizes, and cold-start techniques; online A/B tests report engagement lifts on Home Feed and Related Items. The paper claims a 20% increase in engagement with new items and states that PinFM is deployed to serve more than half a billion users.","tokens_in":16476,"tokens_out":5970,"duration_ms":59973,"significance":"If the results hold, PinFM is a substantial industrial contribution: it demonstrates a viable pretrain-and-fine-tune paradigm for user activity sequences across multiple recommender surfaces, and the engineering innovations (DCAT, int4 quantization, KV-cache sharing) address real serving constraints for a model with a 20-billion-parameter embedding table. The paper reports several negative or neutral results honestly, such as the I2I Fresh Saves decline in Table 7 and the small int4 quantization drop in Section 4.2. The main weaknesses are statistical reporting and an overclaimed cold-start headline; these are fixable and do not invalidate the core engineering/empirical contribution.","major_comments":[{"comment":"The abstract's claim that PinFM leads to 'a 20% increase in engagement with new items' is not supported by the reported data. Table 2 reports relative HIT@3 lifts against a 'w/o PinFM' baseline: the final cold-start variant is +17.72% on HF 28d Save and +12.01% on HF 7d. The 'more than 20% boost' in the text is the arithmetic gap between two PinFM variants (-4.40% for PinFM-cs-none and +17.72% for PinFM-cs-CIR-IDD-GSLT), not a measured 20% increase in user engagement. The online Fresh Saves lift for Home Feed in Table 7 is +5.70%, not 20%. Please rephrase the claim as an offline relative HIT@3 gain or supply online engagement evidence that actually supports a 20% effect.","section":"Abstract and Section 5.1, Table 2"},{"comment":"All offline comparisons report only relative HIT@3 lifts, with no raw hit rates, confidence intervals, standard errors, or significance tests. HIT@3 is a custom metric, and the description of how recommendation groups are formed and how ties are handled is underspecified. Because the headline conclusions (e.g., the 3.76% HF Save lift in Table 1 and the cold-start gains in Table 2) rest entirely on this metric, the paper should report raw values, evaluation sample sizes, and some measure of variability, ideally across repeated runs or held-out time periods.","section":"Section 5.1, Tables 1-6"},{"comment":"The statement that all online results are statistically significant at the 95% confidence level is not substantiated. No p-values, confidence intervals, sample sizes, or multiple-comparison corrections are given, and the I2I Fresh Saves entry is -0.82% while still being asserted as significant. Please describe the A/B setup (randomization unit, experiment duration, metric definitions) and report uncertainty for each lift, including negative results.","section":"Section 5.2, Table 7"},{"comment":"The 'w/o PinFM' baselines are not described in terms of model capacity, training budget, hyperparameter tuning, or feature set. If the baseline ranking models were not trained to a comparable level of effort, all reported relative lifts would be inflated. Please document the baseline training configuration and, where possible, report the performance of the same downstream model with an equally sized sequence encoder trained only on downstream data.","section":"Section 5.1, Tables 1 and 2"},{"comment":"The cold-start techniques (CIR rate of 10%, IDD dropout probabilities of 0.7 and 0.5, age thresholds of 7 and 28 days, and the GSLT variant) appear to have been selected after observing the negative fresh-item effect of PinFM-cs-none, and no held-out validation or repeated evaluation is reported. This creates a selection-on-the-evaluation-metric risk for the Table 2 gains. Please validate the chosen settings on a time-based holdout or report stability across repeated experiments.","section":"Sections 3.2 and 5.1, Table 2"}],"minor_comments":[{"comment":"The variant name 'PinFM-cs-CPR' in the IDD definition appears to be a typo for 'PinFM-cs-CIR'.","section":"Section 3.2, Table 2"},{"comment":"The reported 1.4% drop from eliminating causal masking is not tied to any table or uncertainty estimate; please attach it to the relevant ablation or provide the supporting numbers.","section":"Section 3.3"},{"comment":"The throughput improvements (600% serving, 200% training) lack measurement context; specify hardware, batch sizes, sequence lengths, and the baseline attention implementation against which the gains are computed.","section":"Section 4.1"},{"comment":"The '20B+ parameters' claim should clarify that the embedding tables contribute roughly 20 billion parameters and that the transformer layers are a separate, smaller component.","section":"Abstract and Section 4.2"},{"comment":"The y-axis labels and units for Save and Hide lift are missing, and the '0 iteration' baseline should be explicitly defined in the caption.","section":"Figure 3"},{"comment":"The sentence 'we observed increase an in feed diversity' contains a typo, and the diversity metric is not defined; please add a precise definition.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"This is an industry systems paper with proprietary data and no code release; that is not disqualifying for the venue, but it raises the burden on statistical reporting. The core engineering story is credible, but the abstract and Section 5.1 overstate the cold-start effect, and the offline tables need variance and baseline-effort information before the quantitative claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the system is real and the online results are plausible, but the abstract's 20% cold-start claim does not survive contact with Table 2. That figure is the arithmetic gap between two PinFM variants on a custom HIT@3 proxy (-4.40% to +17.72%), not a measured engagement lift; the online Fresh Saves lift is +5.70%. The abstract should say what the data actually support.\n\nWhat is genuinely good: pretraining a single 20B-parameter ID-based transformer over user action sequences and fine-tuning it into two existing ranking surfaces is a believable instantiation of the pretrain/finetune paradigm for recsys, and the paper does the engineering that makes it non-trivial. DCAT reuses KV caches across candidates, giving a 600% serving throughput gain, and int4 embedding quantization shows statistically neutral online performance. The application ablations (input sequence variants, multi-token and future-token losses, positive action selection, vocab scaling, pretraining duration) are exactly the kind of detail practitioners need. The online A/B on Home Feed and Related Items is genuinely informative: sitewide Saves +1.20%/+0.72%, Fresh Saves +5.70% on HF. The paper is also transparent about having introduced cold-start fixes only after observing a negative fresh-item effect, which is more honest than most.\n\nWhere it is soft. The headline claim is the problem. Table 2 does not compare PinFM to a deployed model for cold-start; it compares PinFM with no cold-start handling to PinFM with three stacked fixes, on HIT@3, with no confidence intervals and no raw rates. The 'more than 20% boost' is therefore an overstatement when the abstract promises a 20% increase in user engagement. I also share the worry that the 'w/o PinFM' baselines are not demonstrated to have had comparable training effort; a single sentence saying they were trained to the same budget and validated would help. Offline tables have no error bars; online significance is asserted at 95% without p-values or any mention of multiple comparisons. The custom HIT@3 metric is reasonable but unvalidated against user-level engagement. The 'first foundation model' claim is too strong given 360Brew and others.\n\nOverall: this is a solid industrial systems paper with a load-bearing but fixable overclaim. A serious referee should see it; I would recommend major revision rather than desk rejection. The authors need to correct the abstract, add uncertainty numbers where possible, describe baseline training, and either drop or reframe the 20% claim. If those are fixed, it is a useful reference for anyone building large sequence models in production recsys.","headline":"The deployed system and online A/B lifts are credible, but the abstract's 20% cold-start claim is unsupported by Table 2 and needs to be reframed before this is publishable.","tokens_in":17056,"tokens_out":2834,"would_cite":true,"duration_ms":29308,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PinFM is a foundation model for user activity sequences: a 20-billion-parameter transformer, pretrained on years of user actions and fine-tuned into downstream ranking models, that improves engagement metrics and cold-start item…","keywords":["recommender systems","user activity sequences","foundation model","pretraining and fine-tuning","sequential recommendation","cold-start items","cross-attention transformer","embedding quantization"],"falsifier":"Train the same downstream ranking models with identical compute and fine-tuning budgets, replacing PinFM's pretrained sequence encoder with a randomly initialized transformer of the same architecture and size; if the HIT@3 and online engagement lifts persist at the same magnitude, the gains are attributable to capacity and fine-tuning rather than to pretrained user-sequence knowledge. A second check is to evaluate the fine-tuned models on a public next-item prediction benchmark: if PinFM underperforms smaller baselines there, the reported gains are surface-specific rather than a property of the pretrained representation.","tokens_in":16031,"feed_emoji":"🎯","tokens_out":6989,"duration_ms":71379,"temperature":0.7,"pith_summary":"PinFM is a foundation model for user activity sequences: a transformer with over 20 billion parameters, pretrained on roughly two years of user actions and then fine-tuned into downstream ranking models. The paper's central claim is that this pretrain-then-finetune recipe works for industrial recommender systems, where a single large sequence model can be reused across multiple applications and surfaces. The authors report that adding PinFM to the Home Feed and Related Items ranking models improves engagement in online A/B tests, including a 5.70% lift in Saves on fresh items on the Home Feed. They also report that engineering the input sequence, such as appending candidate item features, lets the model learn interactions between user history and candidate items, increasing engagement with new items. If those results hold, they show that a foundation-model approach can scale to billion-user traffic while staying within recommender latency and cost budgets.","feed_headline":"Pretrained 20B model lifts fresh saves 5.7% in live A/B test","feed_subtitle":"One shared transformer over two years of user activity fine-tunes into ranking models with neutral serving cost.","key_machinery":"The load-bearing machinery is threefold. The first is the Deduplicated Cross-Attention Transformer (DCAT), which computes the transformer's context pass once per user, stores per-layer key/value caches, and then reuses those caches to score many candidates via cross-attention instead of running full self-attention for every candidate; the paper reports a 600% serving-throughput improvement over FlashAttention-based self-attention. The second is input sequence engineering for fine-tuning: the candidate item is appended to the user-action sequence, optionally with a GraphSAGE candidate embedding summed into the candidate's representation and a learnable token before it (the PinFM-GraphSAGE-LT variant), so the transformer can learn user-candidate interactions through attention. The third is a set of cold-start and efficiency mechanisms: candidate item id randomization during fine-tuning, item-age dependent dropout on module outputs for fresh items, and post-training int4 quantization of the billion-row ID embedding tables, which cuts the embedding table to 31.25% of its original size with statistically neutral online performance. Pretraining losses, an infoNCE next-token loss, a multi-token loss, and a future-token loss, shape the pretrained representations, while a reduced learning rate and optional sequence losses keep the pretrained knowledge intact during fine-tuning.","core_discovery":"On the paper's own terms, the discovery is that a large transformer pretrained with next-token and multi-token prediction objectives over user activity sequences can be coupled to existing ranking models across different applications and improve their engagement metrics. PinFM processes a user's action history into a representation; the candidate item is inserted into the input sequence so that cross-attention can contextualize the history against that candidate. The paper reports offline HIT@3 lifts (for example, a 3.76% relative improvement in Home Feed Save HIT@3 for the best input variant) and statistically significant online A/B gains: Home Feed sitewide Saves +1.20%, Home Feed Fresh Saves +5.70%, and Related Items sitewide Saves +0.72%. It further claims that the cold-start techniques, candidate item randomization and item-age dependent dropout, turn an offline Fresh Saves deficit of about -4% into a +17% lift, and that the Deduplicated Cross-Attention Transformer plus quantized embeddings keep the added serving cost neutral. The paper therefore asserts an end-to-end paradigm: one pretrained sequence model, shared across applications, can be fine-tuned per surface, handle items unseen at pretraining time, and run online at billion-user scale.","pith_inferences":["A natural extension would be to apply the DCAT caching trick to any ranking stage where the user context is shared across a large candidate list, such as retrieval reranking, where the same ratio of one user sequence to many candidates holds and the same KV-reuse pattern could cut cost.","The cold-start recipe suggests a general principle: randomizing the identity embedding during training and regularizing the module's output for fresh items decouples overall gains from fresh-item losses; this is testable on other id-based ranking models beyond the two surfaces reported.","Because the reported lifts are relative to internal baselines and evaluated with HIT@3, the paper does not establish that PinFM would beat smaller sequence models on public benchmarks; testing the same fine-tuning recipe on a public sequential-recommendation dataset would clarify how much of the gain comes from pretraining scale versus platform-specific features.","The input-sequence ablations imply that the model's understanding of a candidate depends on where and how the candidate appears in the sequence; varying the position and number of candidate tokens, rather than only appending one, is a dimension the paper leaves unexplored."],"forward_implications":["A single pretrained sequence model can be amortized across multiple recommender surfaces, avoiding the cost of training a large sequence model separately for each application.","Candidate items can be folded into the user sequence, so the same foundation model learns user-candidate interactions by input construction rather than by a separate interaction module.","Fresh-item performance can be recovered with candidate-id randomization and age-dependent dropout, which the paper demonstrates by moving Home Feed Fresh Saves offline from negative to strongly positive and online to +5.70%.","The DCAT reuse pattern (one user context scored against many candidates) plus quantized embeddings makes a 20-billion-parameter model servable within recommender latency budgets, supporting deployment to over half a billion users.","Fine-tuning is required: without it the lifts vanish, which the paper reads as evidence that pretraining alone does not transfer directly to ranking objectives."],"supporting_citations":[{"why":"Large sequential transducer for generative recommendations that motivates the scale and serves as an architecture comparison point.","marker":"[31]"},{"why":"Existing transformer-based real-time sequence baseline that the online A/B tests compare against.","marker":"[27]"},{"why":"Ranking model details for Home Feed and Related Items, the downstream fine-tuning and evaluation framework.","marker":"[28]"},{"why":"Shows pretrained ID embeddings transfer to downstream models, motivating the ID-based sequence representation.","marker":"[8]"},{"why":"InfoNCE contrastive objective used to train the model over a billion-item vocabulary.","marker":"[19]"},{"why":"FlashAttention baseline against which the DCAT kernel achieves the reported throughput improvement.","marker":"[4]"},{"why":"FBGEMM quantization kernels used for post-training int4 and int8 embedding compression.","marker":"[14]"},{"why":"GraphSAGE candidate embeddings summed into the sequence in the best-performing input variant.","marker":"[30]"}],"fun_headline_variants":["20B user-activity transformer lifts fresh saves 5.7% online","Pretrained 20B model: fresh saves +5.7% live","Foundation model for activity sequences scores 5.7% fresh-save gain","One 20B transformer improves fresh saves 5.7% across apps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on the assumption that the HIT@3 metric (whether the top three items in a same-time group received the corresponding user action) faithfully reflects recommendation quality and that the \"without PinFM\" baselines were trained as hard as the PinFM-augmented models; if the baselines are undertrained or the metric is gameable, the reported relative lifts would not establish the central claim.","fun_headline_variants_meta":{"raw":{"variants":["20B user-activity transformer lifts fresh saves 5.7% online","Pretrained 20B model: fresh saves +5.7% live","Foundation model for activity sequences scores 5.7% fresh-save gain","One 20B transformer improves fresh saves 5.7% across apps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001328,"raw_usage":{"total_tokens":5450,"prompt_tokens":1038,"completion_tokens":4412,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":4329}},"tokens_in":654,"tokens_out":4412,"duration_ms":34755,"temperature":1.0,"reasoning_tokens":4329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:41:16.782370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same downstream ranking models with identical compute and fine-tuning budgets, replacing PinFM's pretrained sequence encoder with a randomly initialized transformer of the same architecture and size; if the HIT@3 and online engagement lifts persist at the same magnitude, the gains are attributable to capacity and fine-tuning rather than to pretrained user-sequence knowledge. A second check is to evaluate the fine-tuned models on a public next-item prediction benchmark: if PinFM underperforms smaller baselines there, the reported gains are surface-specific rather than a property of the pretrained representation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large sequential transducer for generative recommendations that motivates the scale and serves as an architecture comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ranking model details for Home Feed and Related Items, the downstream fine-tuning and evaluation framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows pretrained ID embeddings transfer to downstream models, motivating the ID-based sequence representation."}],"review_version":1}