REVIEW 4 major objections 4 minor 1 cited by
A billion-user video recommender can generate natural-language interest personas for every user in real time, and doing so measurably increases watch time and exploration.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 11:44 UTC pith:TS6D5DAJ
load-bearing objection Real industrial demonstration of real-time LLM personas with a small but significant A/B lift; the causal story hinges on an unmeasured text-to-item grounding step, and the live test used a suboptimal clustering variant. the 4 major comments →
LLM-Based User Personas for Recommendations at Scale
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a distilled LLM, prompted with semantically clustered watch history, can produce a dual-purpose persona in one inference call: a summarized list of the user's existing interests (exploitation) plus novel but related exploration topics (exploration). Those text interests are then grounded into candidate videos by restricting a sequential transformer's nearest-neighbor search to items semantically related to the generated text. The paper reports that this pipeline, running asynchronously so that no user request waits on the LLM, produced statistically significant lifts in watch time and active users in a live 30-day A/B test, plus a +0.04% lift in engaged topics and +
What carries the argument
The load-bearing mechanism is a hierarchical planning loop: a high-level language policy (the LLM persona) that outputs free-text interests, and a low-level item policy (a sequential transformer with restricted nearest-neighbor search) that maps those interests to concrete video candidates. To make the high level cheap enough for online use, the system distills a large teacher model's reasoning into a small student model, quantizes it, and serves it asynchronously: if a user's persona is missing or stale, a background job regenerates it without blocking the current request. The prompt itself is the other key piece—grouping the user's watch history into semantic clusters before asking for sum
Load-bearing premise
The entire case rests on the assumption that a free-form natural-language interest, when used to constrain nearest-neighbor item search, actually retrieves items that match what the LLM meant—an alignment the paper never measures directly.
What would settle it
Run the same live system with the LLM-generated interests replaced by random or scrambled text of similar length fed through the same restricted nearest-neighbor search; if the watch-time and active-user gains persist, then the semantic content of the personas is not the active ingredient. Alternatively, have human raters judge whether items retrieved from a persona are topically aligned with that persona's text; near-chance alignment would falsify the grounding step.
If this is right
- Natural-language personas can be embedded in the live serving path of a billion-user recommender without violating latency budgets, enabling real-time adaptation to shifting interests.
- Exploration interests, not just summaries, are responsible for the novelty gains: exploration-retrieved items are 13.6% more likely to be watched once shown, even though they receive far fewer impressions.
- The benefit concentrates among casual users, suggesting LLM personas are particularly good at inferring preferences from sparse interaction histories.
- User-facing interpretability is feasible: surveyed users largely recognized their own interests in the generated labels, and preferred them to an entity-based baseline.
Where Pith is reading between the lines
- The paper's causal story depends on the grounding step (free-text interest to semantically related items) being accurate, but grounding quality is never measured directly; final A/B metrics are the only evidence, so an alternative explanation (e.g., the system just adds noise that happens to help) cannot be fully ruled out.
- The random sampling of one summarized and one exploration interest per request is a deliberate exploration policy; a natural next step is to replace it with a predicted-utility selection, which could amplify the small gains.
- The approach should transfer to other domains with rich text metadata (e.g., news, e-commerce, podcasts), but the specific cluster-then-prompt recipe would need re-tuning since title quality and noise patterns differ.
- The +0.04% watch-time lift, while statistically significant, is small; a plausible reading is that the downstream ranking model dilutes the persona signal, so the method's full potential may require retraining the ranker to incorporate persona features.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an industrial framework for generating natural-language user interest personas with LLMs at live serving time in a large-scale video recommendation platform. The method combines two persona types: summarized interests (exploitation) and exploration interests (novel but related topics). To meet billion-user latency/cost constraints, it uses semantic clustering of watch history, knowledge distillation from Gemini 1.5 Pro to smaller student models, asynchronous background generation, and quantization. Personas are grounded into item space via a sequential transformer with restricted nearest-neighbor search. The paper reports offline BLEURT-based representation studies, distillation performance metrics, a user survey, and a 30-day live A/B test with statistically significant (p<0.05) gains of +0.04% watch time and +0.03% active users, plus exploration metrics.
Significance. If the live results are trustworthy, this is a meaningful industrial demonstration: real-time LLM-based persona generation at billion-user scale with measurable viewer-value gains. The live A/B test is a genuine external validation that does not rely on the teacher model, and the user survey adds qualitative support. The architecture is clearly described and the cost-efficiency discussion is a useful contribution. However, the paper's internal consistency and causal attribution have gaps that need to be addressed before the central claim is fully supported.
major comments (4)
- [§5.4.1 vs §3.2] The offline study (Figure 3(a)) concludes that semantic-based clusters generate higher-quality personas than embedding-based clusters, yet the live experiment (§5.4.1) explicitly uses embedding-based clustering. No explanation is given for this mismatch. Since the headline claim rests on the live A/B, the reader cannot tell whether the deployed system reflects the paper's recommended design or why the apparently superior representation was not used. Please clarify the reason and, if possible, provide a live comparison or an offline analysis showing the impact of this choice.
- [§3.1, §5.4.2] The restricted nearest-neighbor search that maps persona text to items is never directly evaluated. The paper does not measure whether retrieved items are semantically related to the LLM-generated persona. BLEURT and user surveys assess the summaries in the abstract, and the live A/B reports final user metrics, but none of these establish that the persona's text content drives the observed gains. The +0.04% watch time and +0.03% active-user lifts could plausibly arise from the random sampling of interests introducing exploration diversity independent of persona semantics. Please add a grounding-quality evaluation (e.g., human relevance ratings, embedding similarity between retrieved items and the persona, or a random-text-interest counterfactual).
- [§5.3, §5.2] Section 5.3 does not state which model (teacher or distilled student) generated the interests rated by users. If the survey used the teacher, it does not validate the deployed Gemini Nano student; if it used the student, that should be stated explicitly. Likewise, the distillation metrics in Table 2 use teacher-generated summaries as the BLEURT reference and an LLM autorater comparing against the teacher, so they measure fidelity to the teacher, not absolute quality. The live A/B is the independent evidence for the student's value, but the survey's evidential role is ambiguous. Please clarify the model used and frame Table 2 as teacher-fidelity metrics.
- [§5.4.2] The exploration decomposition reports a 40.91% exposure gap and a 13.6% higher conditional watch probability for exploration items, but it does not provide confidence intervals or p-values for these quantities. Moreover, the higher conditional watch probability could be a selection effect—exploration items that survive rankers may be unrepresentative—or a novelty effect rather than a reflection of persona-relevant content. Please report statistical uncertainty and discuss these alternative explanations.
minor comments (4)
- [References] Reference [15] and [16] appear to be the same work (one arXiv, one RecSys), causing duplication. Also reference [22] has an arXiv ID but is formatted differently from other arXiv entries. Please clean up the bibliography.
- [Figure 4] The caption says the X-axis 'represents the time'; it should be 'represents time' or 'represents the experiment date'. Also, please ensure error bars or confidence intervals are shown in the figure.
- [Table 2] For epoch 0, BLEURT and Creativity are N/A for Nano due to near-zero IFR. Consider explaining this in the caption more explicitly, since the current note is easy to overlook.
- [§4.1.1] Insight 3 states that performance plateaus when scaling from Pro to Ultra; this is mentioned as motivation for distillation, but no numbers are given. A brief quantitative comparison would strengthen the claim.
Circularity Check
No significant circularity: the central claim is grounded by an external live A/B test and user surveys, and the distillation evaluation is explicitly framed as imitation fidelity.
full rationale
The paper's derivation chain is: structured user history -> LLM persona generation -> distillation -> restricted nearest-neighbor item retrieval -> live A/B test. The only evaluation that compares student outputs to teacher outputs is Section 5.2, where BLEURT and Creativity scores are explicitly used to measure how well the student preserves the teacher's summaries and exploration interests. This is appropriately framed as distillation fidelity, not as independent evidence of persona quality. Independent support comes from Section 5.1 (summaries evaluated against clicked topics), Section 5.3 (user surveys rating label accuracy and preference), and Section 5.4 (a 30-day live A/B test against a production baseline without personas). The A/B metrics (watch time, active users, engaged topics) are external behavioral outcomes, not quantities constructed from the model's own outputs. No fitted parameter is renamed as a prediction, and no uniqueness theorem or self-citation is used to force the central conclusion. Self-citations to prior work [3, 22, 23, 24, 25] describe adopted building blocks (semantic clustering, hierarchical planning paradigm), but the effectiveness claim rests on the new live experiment. The paper's weak grounding analysis (text-to-item semantic mapping) is a correctness risk or alternative-explanation concern, not a circularity: the paper does not define the target result in terms of its inputs. Therefore no specific circular step can be exhibited, and the appropriate score is 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- Semantic clustering similarity threshold =
unknown
- Number of clusters (Num_Groups) and videos sampled per cluster =
unknown
- Distillation checkpoint epoch 26.20 =
26.20
- Sampling policy: one summarized + one exploration interest per request =
1+1
axioms (4)
- domain assumption Salient-term semantic clusters of watched videos faithfully represent user interests
- domain assumption Teacher Gemini 1.5 Pro outputs are high-quality ground truth for summarization and exploration
- domain assumption BLEURT and the LLM autorater's creativity score are valid proxies for persona quality and user value
- domain assumption Restricted nearest-neighbor search can ground free-form LLM text interests into relevant video candidates
read the original abstract
Large Language Models (LLMs) offer unprecedented potential for enhancing recommendation systems through their world knowledge and reasoning capabilities. However, existing approaches often rely on structured IDs or offline processing, limiting semantic richness, real-time adaptability, and user-facing interpretability. In this paper, we introduce a novel framework that enables real-time generation of LLM-based user interest personas for a large-scale commercial video recommendation platform. Our method generates natural-language user interest personas that address the exploitation-exploration trade-off by combining the summarization of existing interests with novel topics, directly during serving. To overcome the computational challenges of online LLM inference at a billion-user scale, we design a cost-efficient architecture leveraging knowledge distillation, asynchronous inference, and input optimization via semantically clustered video representations. Extensive offline evaluations, user studies, and live A/B tests demonstrate significant improvements in viewer value. This work bridges the gap between high-level semantic understanding and industrial-scale recommendation, paving the way for more dynamic, explainable, and satisfying personalized experiences.
Figures
Forward citations
Cited by 1 Pith paper
-
RecGPT-V3 Technical Report
A stateful LLM recommender with memory, text-plus-Semantic-ID grounding, and latent reasoning reports higher Taobao engagement and sales at ~52% lower serving compute than its predecessor.
Reference graph
Works this paper leans on
-
[1]
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. InProceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). ACM, 1007–1014. https://doi. org/10.1145/3604915.3608857
arXiv 2023
-
[2]
Moumita Bhattacharya, Vito Ostuni, and Sudarshan Lamkhede. 2024. Joint Mod- eling of Search and Recommendations Via an Unified Contextual Recommender (UniCoRn). arXiv:2408.10394 [cs.IR] https://arxiv.org/abs/2408.10394
Pith/arXiv arXiv 2024
-
[3]
Konstantina Christakopoulou, Alberto Lalama, Cj Adams, Iris Qu, Yifat Amir, Samer Chucri, Pierce Vollucci, Fabio Soldo, Dina Bseiso, Sarah Scodel, Lucas Dixon, Ed H. Chi, and Minmin Chen. 2023. Large Language Models for User Interest Journeys. arXiv:2305.15498 [cs.CL] https://arxiv.org/abs/2305.15498
Pith/arXiv arXiv 2023
-
[4]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. InProceedings of the 10th ACM Conference on Recommender Systems(Boston, Massachusetts, USA)(RecSys ’16). Association for Computing Machinery, New York, NY, USA, 191–198. https://doi.org/10.1145/ 2959100.2959190
arXiv 2016
-
[5]
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Mur- phy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. InProceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, New York, USA)(KDD ’14). Association fo...
arXiv 2014
-
[6]
Francesco Fabbri, Gustavo Penha, Edoardo D’Amico, Alice Wang, Marco De Nadai, Jackie Doremus, Paul Gigioli, Andreas Damianou, Oskar Stål, and Mounia Lalmas
-
[7]
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2023. Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). arXiv:2203.13366 [cs.IR] https://arxiv.org/abs/ 2203.13366
Pith/arXiv arXiv 2023
-
[8]
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. InInternational Conference on Machine Learning. https: //arxiv.org/abs/1908.10396
Pith/arXiv arXiv 2020
-
[9]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk
-
[10]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. InProceedings of the 22nd ACM International Conference on Information & Knowledge Management(San Francisco, California, USA)(CIKM ’13). Association for Computing Machinery, New York, NY, USA, ...
arXiv 2013
-
[11]
Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Recom- mendation. arXiv:1808.09781 [cs.IR] https://arxiv.org/abs/1808.09781
Pith/arXiv arXiv 2018
-
[13]
Yiqun Liu, Kaushik Rangadurai, Yunzhong He, Siddarth Malreddy, Xunlong Gui, Xiaoyi Liu, and Fedor Borisyuk. 2021. Que2Search: Fast and Accurate Query and Document Understanding for Search at Facebook. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining(Virtual Event, Singapore)(KDD ’21). Association for Computing Machinery...
arXiv 2021
-
[16]
Changping Meng, Hongyi Ling, Jianling Wang, Yifan Liu, Shuzhou Zhang, Dapeng Hong, Mingyan Gao, Onkar Dalal, Ed Chi, Lichan Hong, Haokai Lu, and Ningren Han. 2025. Balancing Fine-tuning and RAG: A Hybrid Strat- egy for Dynamic LLM Recommendation Updates. InProceedings of the Nine- teenth ACM Conference on Recommender Systems (RecSys ’25). ACM, 919–922. ht...
arXiv 2025
-
[17]
Anand Rajagopalan, Fabio Vitale, Danny Vainstein, Gui Citovsky, Cecilia M Procopiuc, and Claudio Gentile. 2021. Hierarchical Clustering of Data Streams: Scalable Algorithms and Approximation Guarantees. InProceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (...
2021
-
[18]
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. arXiv:2004.04696 [cs.CL] https://arxiv.org/ abs/2004.04696
Pith/arXiv arXiv 2020
-
[19]
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional En- coder Representations from Transformer. arXiv:1904.06690 [cs.IR] https: //arxiv.org/abs/1904.06690
Pith/arXiv arXiv 2019
-
[20]
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530(2024)
Pith/arXiv arXiv 2024
-
[21]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762
Pith/arXiv arXiv 2023
-
[22]
Chi, Lichan Hong, and Haokai Lu
Haoting Wang, Jianling Wang, Hao Li, Fangjun Yi, Mengyu Fu, Youwei Zhang, Yifan Liu, Liang Liu, Minmin Chen, Ed H. Chi, Lichan Hong, and Haokai Lu. 2025. Serendipitous Recommendation with Multimodal LLM. (2025). arXiv:2506.08283 [cs.IR] https://arxiv.org/abs/2506.08283
arXiv 2025
-
[23]
Chi, Lichan Hong, Ningren Han, and Haokai Lu
Jianling Wang, Yifan Liu, Yinghao Sun, Xuejian Ma, Yueqi Wang, He Ma, Zhengyang Su, Minmin Chen, Mingyan Gao, Onkar Dalal, Ed H. Chi, Lichan Hong, Ningren Han, and Haokai Lu. 2025. User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems. arXiv:2504.05522 [cs.IR] https://arxiv.org/abs/2504.05522
Pith/arXiv arXiv 2025
-
[24]
Jianling Wang, Haokai Lu, Yifan Liu, He Ma, Yueqi Wang, Yang Gu, Shuzhou Zhang, Ningren Han, Shuchao Bi, Lexi Baugher, et al. 2024. Llms for user interest exploration in large-scale recommendation systems. InRecSys
2024
-
[25]
Jianling Wang, Haokai Lu, Yifan Liu, He Ma, Yueqi Wang, Yang Gu, Shuzhou Zhang, Ningren Han, Shuchao Bi, Lexi Baugher, Ed Chi, and Minmin Chen. 2024. LLMs for User Interest Exploration in Large-scale Recommendation Systems. arXiv:2405.16363 [cs.IR] https://arxiv.org/abs/2405.16363
Pith/arXiv arXiv 2024
-
[26]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903
Pith/arXiv arXiv 2023
-
[27]
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-Based Recommendation with Graph Neural Networks.Proceedings of the AAAI Conference on Artificial Intelligence33, 01 (July 2019), 346–353. https: //doi.org/10.1609/aaai.v33i01.3301346
-
[28]
Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yan- bin Lu. 2023. PALR: Personalization Aware LLMs for Recommendation. arXiv:2305.07622 [cs.IR] https://arxiv.org/abs/2305.07622
Pith/arXiv arXiv 2023
-
[29]
Hamed Zamani and W. Bruce Croft. 2018. Joint Modeling and Optimization of Search and Recommendation. arXiv:1807.05631 [cs.IR] https://arxiv.org/abs/ 1807.05631
Pith/arXiv arXiv 2018
-
[30]
Haiyuan Zhao, Lei Zhang, Jun Xu, Guohao Cai, Zhenhua Dong, and Ji-Rong Wen
-
[2016]
arXiv:1511.06939 [cs.LG] https://arxiv.org/abs/1511.06939
Session-based Recommendations with Recurrent Neural Networks. arXiv:1511.06939 [cs.LG] https://arxiv.org/abs/1511.06939
-
[2023]
InProceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23)
Uncovering User Interest from Biased and Noised Watch Time in Video Recommendation. InProceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23). ACM, 528–539. https://doi.org/10.1145/3604915.3608797
-
[2025]
InProceedings of the Nineteenth ACM Conference on Recommender Systems (RecSys ’25)
Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge. InProceedings of the Nineteenth ACM Conference on Recommender Systems (RecSys ’25). ACM, 1181–1186. https://doi.org/10.1145/3705328.3759305
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.