{"id":"ba9dddc6-5bc1-4960-ab7a-e84bb51fb143","arxiv_id":"2602.12354","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A transformer-based sequential recommender, Feed SR, improved LinkedIn Feed time spent by 2.10% in an online A/B test and now serves the majority of Feed traffic.","lead":"LinkedIn replaced its production Feed ranker with a transformer-based sequential model that processes a member's last 1,000 post interactions and reports +2.10% time spent in online tests. The paper is a detailed industrial case study of making sequential transformers work under strict latency and throughput constraints at 1.2-billion-member scale.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Position-debiasing validity is the linchpin of the A/B lift; without calibration evidence across positions, +2.10% time spent may be a positional artifact.","rationale":"The reader's weakest assumption zeroes in on position debiasing as the load-bearing premise for the online lift. I agree. The paper's architecture and engineering are credible, and the deployment claim is plausible, so this is not a question of fraud or internal inconsistency. But the single quantitative result that carries the central claim is Table 5's +2.10% time-spent lift, and the only argument that this lift is about ranking quality rather than positional exposure is the debiasing paragraph in Section 4.6.1. That paragraph is thin: IPW and explicit position logit offsets are both used without discussing how they interact, and the fixed position-5 scoring is described only as 'well-calibrated' with no supporting calibration curves. Since the position offset is constant across candidates at serving, the ranking itself is unaffected by the offset; the risk is that training-time debiasing failures contaminate the learned representations. The lack of confidence intervals and the missing +3.52% contribution metric in the results section further weaken the evidentiary base. A stratified analysis of the A/B logs by served position is a concrete, feasible check that would distinguish an artifact from a genuine relevance gain. The reader's conditional verdict is appropriate; my stress test does not move it.","tokens_in":15467,"tokens_out":8194,"duration_ms":81354,"concrete_test":"From the A/B logs, stratify the time-spent lift by the actual feed position at which items were served (e.g., positions 1–5, 6–10, 11–20, 20+). If the +2.10% gain is concentrated in top positions and vanishes or reverses at lower positions, the result is consistent with a positional-exposure artifact rather than improved relevance. If the lift is robust across lower position buckets, the position-debiasing concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests almost entirely on the online A/B result in Table 5 (+2.10% time spent). The only mechanism offered to ensure this lift reflects relevance rather than position bias is Section 4.6.1: IPW plus learned position logit offsets, with online scoring at position 5. This is not demonstrated. Combining IPW and explicit position logit offsets risks double-counting position bias; the paper gives no analysis of their interaction, no propensity diagnostics, and no confidence intervals for Table 5. If the debiasing is miscalibrated, the model may learn to exploit historical positional exposure (e.g., favoring items that often appeared at high positions) rather than intrinsic relevance. At serving, scoring all candidates at position 5 makes the position offset a constant, so it does not affect ranking; it also cannot correct biases already baked into the learned representations during training. Furthermore, the abstract claims a +3.52% contribution lift that is never reported in the results section, leaving the time-spent metric as the sole quantitative support for the headline claim. A stratified analysis of the A/B logs by served position would directly test whether the lift is due to improved relevance or to positional exposure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Feed Sequential Recommender (Feed SR), a transformer-based sequential ranking model that replaces the DCNv2-based production ranker for LinkedIn Feed. The model interleaves post and action tokens, uses RoPE, late-fused context features, an MMoE head, IPW plus learned position logit offsets, and a range of training/inference optimizations. Offline ablations report AUC improvements across head architectures, features, loss weighting, and scaling behaviour. Online A/B results in Table 5 claim +2.10% time spent overall, with the largest gains for active members and neutral for new members. Section 9 states Feed SR is now the majority member experience. The abstract additionally claims +3.52% like/comment/reshares, which does not appear in Section 7.","tokens_in":15751,"tokens_out":4503,"duration_ms":41001,"significance":"If the online result is a clean measurement, the paper is a valuable industrial case study showing that a transformer sequential model using roughly 20% of production features can outperform a DCNv2 ranker at LinkedIn scale. The paper's strengths include detailed serving optimizations (SRMIS kernel, shared context batching), compute-matched HSTU comparisons, scaling-law analyses along multiple axes, and honest deployment lessons. The main risk is that the headline lift is a single point estimate with no confidence interval, and the position-debiasing mechanism is asserted rather than validated. These issues are addressable with additional analysis, so the paper merits major revision rather than rejection.","major_comments":[{"comment":"The abstract claims +3.52% like/comment/reshares (LCR), but Table 5 in Section 7 reports only time-spent metrics; no LCR result appears anywhere in the results section. If LCR was measured, include it with a confidence interval; if not, the abstract overstates the findings. Additionally, the +2.10% time-spent figure is a single point estimate with no CI, significance level, or segment-level variance. Given that Section 7 breaks out DAU/WAU/MAU/New, reporting standard errors or confidence intervals would materially strengthen the central claim.","section":"Section 7 / Abstract"},{"comment":"The validity of the online lift rests on the assertion that scoring at position 5 with IPW and learned position logit offsets removes position bias. The paper gives no diagnostics: no propensity model calibration or coverage check, no analysis of possible double-counting between IPW and explicit position logits, and no stratified online results by served position. Since at serving the position-5 logit offset is constant, it cannot correct position bias already learned from historical exposure; only IPW can, and that is not validated. A stratified analysis of A/B logs by served position, or an offline holdout evaluation with position randomized, would directly test whether the +2.10% reflects relevance rather than positional exposure.","section":"§4.6.1, Table 5"},{"comment":"In-session leakage is mitigated by randomizing item order within sessions, but during serving the model receives chronological order. This train/serve skew is acknowledged but its effect on the reported online gain is not quantified. The paper states that both randomization and attention masking resolve overfitting; if both work, reporting offline AUC for both and an online A/B for the masking variant (or a justification for why randomization does not hurt) would address the concern that the deployed model is trained on a distribution that does not match serving.","section":"§4.7"}],"minor_comments":[{"comment":"'1017 FLOPs' should read '10^17 FLOPs' (the superscript is missing).","section":"§4.2.2"},{"comment":"'LLama' should be 'LLaMA'.","section":"§5.1"},{"comment":"'from and infrastructure perspective' is a typo; should be 'from a modeling and an infrastructure perspective'.","section":"§8.3"},{"comment":"Figures 11 and 12 have y-axis values omitted 'due to legal reasons'; this limits interpretability. Consider plotting normalized axes or explaining the omission in the text.","section":"Appendix D"},{"comment":"The claim that early online experiments show gains beyond those reported is unsupported by any numbers. Either include the results or remove the claim.","section":"§4.6.2"},{"comment":"The feature transforms f_j are not specified as learned or fixed. Clarify which transforms are learned embeddings, log transforms, or identity mappings.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"This is an industrial systems paper whose central evidence is the online A/B lift. The abstract/body inconsistency on LCR is the most immediate editorial issue, and the lack of position-bias diagnostics is the main technical gap. I see no grounds for questioning author integrity; the issues are standard measurement-reporting gaps. The paper would be a strong fit for the venue once the A/B analysis is made more rigorous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: send it to a serious referee. This is a genuine deployment paper, not a new method paper, and its value is in the engineering transparency and the scale of the deployment. A transformer-based sequential ranker replaced a DCNv2 production ranker for the majority of LinkedIn Feed traffic for over three months. That alone is evidence the central claim is probably true. The paper is honest about what it builds on: it says the architecture is based on [30], and it reports negative results with LLM-Ranker and TransAct rather than sweeping them aside.\n\nWhat is actually new is mostly engineering. The SRMIS flash-attention kernel that supports the shared-context multi-candidate mask with two scalar parameters is a useful contribution. The CPU optimizations (225x history parsing, 50x sparse-to-dense) and the training optimizations are concrete and believable. The late-fusion vs early-fusion decision is well motivated and tested offline and online. The energy comparison is a nice touch.\n\nThe soft spots are in the online evidence. Table 5 is a single point estimate with no confidence interval, no significance test, and no breakdown beyond member segments. The abstract version I saw in the submission only claims +2.10% time spent; the reader's copy also has +3.52% LCR, which never appears in the results table. If that number is in the submitted abstract, it needs to be in the table or removed. That is a must-fix, but not a fatal flaw.\n\nThe position-debiasing concern is real but I would not call it load-bearing. IPW plus learned position logit offsets could double-count, and scoring at position 5 makes the offset constant at serving, so the debiasing has to happen during training. The paper gives no propensity diagnostics. But the A/B is end-to-end against the production ranker; if the debiasing were badly miscalibrated, I would expect the offline/online score-discrepancy pipeline they describe in 8.2 to have caught some of it. Still, a stratified analysis by served position would settle it and should be requested.\n\nThe in-session leakage fix is the other thing I would push on. Randomizing session order during training removes leakage but creates a train/serve mismatch. They say masking also fixed the overfitting but was slower; they chose speed. That is a reasonable engineering tradeoff, but they should quantify the cost.\n\nOverall: this paper is for practitioners building large-scale ranking systems. It deserves a serious referee. I would send it to the industry track of RecSys or a similar venue, with requests for confidence intervals, a reconciled abstract, and a position-stratified analysis.","headline":"Credible industrial deployment study with real engineering value; the online lift is under-reported and position debiasing needs a closer look, but this deserves a serious referee.","tokens_in":16340,"tokens_out":4805,"would_cite":true,"duration_ms":46106,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer that ranks LinkedIn's Feed by interleaving posts and member actions beat the production DCNv2 ranker, lifting time spent by 2.10% in online A/B tests.","keywords":["sequential recommendation","feed ranking","transformer","late fusion","position debiasing","inverse propensity weighting","multi-task learning","production recommender system"],"falsifier":"Run an online A/B test where the same trained Feed SR model is served with the scoring position set to 1 and then to 10 instead of 5. If the position-debiasing is correct, the time-spent lift relative to the production ranker should remain roughly constant across these scoring positions; if the measured lift shifts significantly with scoring position, the reported +2.10% gain is not a clean measure of ranking quality.","tokens_in":15356,"feed_emoji":"📈","tokens_out":2942,"duration_ms":29712,"temperature":0.7,"pith_summary":"This paper claims that a decoder-only transformer trained on sequences of interleaved Feed posts and member actions can replace a feature-heavy DCNv2-based production ranker at LinkedIn scale. With only about 20% of the production feature set, the model, called Feed SR, achieved +2.10% time spent and +3.52% likes, comments, or reshares in online A/B tests, and now serves the majority of LinkedIn's Feed traffic. The authors argue that clean position debiasing, late fusion of contextual features, and careful training techniques make sequential modeling practical under strict latency and throughput constraints. If true, this shows that hand-engineered feature transforms in large-scale ranking can be largely substituted by learned sequence models.","feed_headline":"Sequential transformer lifts LinkedIn Feed time spent by 2.10%","feed_subtitle":"Model uses 20% of the old feature set and now ranks the majority of LinkedIn Feed traffic.","key_machinery":"The central object is the interleaved sequence of post and action tokens fed into a decoder-only transformer with a causal attention mask: each historical post is represented by compact feature embeddings, and multi-hot action vectors are projected into the same dimension, so the sequence becomes [post1, action1, post2, action2, ...]. This transformer encodes the member's evolving interests; its output is concatenated with late-fused context features (candidate popularity, dwell-time buckets, viewer–actor affinity) and passed to an MMoE head for multi-task prediction. Position debiasing is handled by learning per-position logit offsets for the top-60 feed positions and scoring candidates at","core_discovery":"Feed SR is a sequential recommender that encodes a member's recent 1000 impressions as an interleaved sequence of post representations and action representations, processes it with a causal transformer using rotary positional embeddings, then late-fuses candidate-conditional numeric features and passes the result through a multi-gate mixture-of-experts head to predict dwell, like, comment, and share actions. Compared to the existing production DCNv2-based ranker, Feed SR improves time spent by +2.10% overall and up to +2.38% for daily active users in online A/B tests, while using a substantially reduced feature set. The model has been serving the majority of LinkedIn Feed traffic for over th","pith_inferences":["If the position-debiasing heuristic (learned offsets plus scoring at position 5) is genuinely robust, the same fixed-position scoring trick could be transferable to other feed-ranking systems that suffer from position bias, without needing online position-aware serving.","The paper's success with a 20% feature set suggests that many cross-features in traditional DLRMs may be redundant when a transformer can learn temporal interaction patterns directly from behavioral sequences.","The failure of the LLM-based ranker, despite promising offline results, points to a practical lesson: token efficiency in encoding user history is a decisive factor for industrial sequential recommenders, since text-based history encoding became prohibitively expensive.","The scaling-law result, where sequence length scales more reliably than depth or embedding dimension, could motivate future work on history-length extension (e.g., using more than 1000 impressions or multi-year histories) rather than purely increasing model depth."],"forward_implications":["Sequential models can replace many manually engineered history transforms in feed ranking, cutting the feature set to about 20% while improving engagement.","Late fusion of candidate-conditional features is nearly as accurate as early fusion but reduces training time by about 12%, and also simplifies online serving by reducing history storage and feature-fetch overhead.","Scoring all candidates in a single forward pass with a shared context (via a custom attention mask and CUDA kernel) makes transformer inference practical at production scale, giving an 80x speedup on the transformer forward pass.","Increasing sequence length is the most consistent scaling axis for this model, suggesting that longer interaction histories are the most reliable path to further quality gains.","Learned absolute position embeddings cause score instability due to highly skewed sequence-length distributions, while RoPE keeps predicted scores stable and improves offline AUC."],"fun_headline_variants":["LinkedIn's sequential recommender boosts feed time spent 2.10%","Transformer ranker lifts LinkedIn feed engagement with fewer features","Feed SR: sequential transformer outperforms DCNv2 in live A/B test","How a causal transformer now ranks most LinkedIn feed impressions","Industrial-scale sequential recommender gains 2.10% dwell time at LinkedIn"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the learned position logit offsets plus scoring at position 5 fully remove position bias, so the reported +2.10% time-spent lift reflects genuine relevance gains rather than a calibration artifact.","fun_headline_variants_meta":{"raw":{"variants":["LinkedIn's sequential recommender boosts feed time spent 2.10%","Transformer ranker lifts LinkedIn feed engagement with fewer features","Feed SR: sequential transformer outperforms DCNv2 in live A/B test","How a causal transformer now ranks most LinkedIn feed impressions","Industrial-scale sequential recommender gains 2.10% dwell time at LinkedIn"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":2923,"prompt_tokens":665,"completion_tokens":2258,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":2166}},"tokens_in":409,"tokens_out":2258,"duration_ms":14433,"temperature":1.0,"reasoning_tokens":2166,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:49:14.133918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an online A/B test where the same trained Feed SR model is served with the scoring position set to 1 and then to 10 instead of 5. If the position-debiasing is correct, the time-spent lift relative to the production ranker should remain roughly constant across these scoring positions; if the measured lift shifts significantly with scoring position, the reported +2.10% gain is not a clean measure of ranking quality.","supporting_citations":[],"review_version":1}