{"id":"d211eb11-02e1-43fe-b356-157ac05848f3","arxiv_id":"2611.16795","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"KuaiRecV2 provides a 30-day, 124k-user, 24M-item industrial benchmark for evaluating generative recommenders on continual learning, token-level diversity, and multi-task performance.","lead":"KuaiRecV2 is a large-scale benchmark dataset and evaluation pipeline for recommender systems, built from 30 days of Kuaishou user interactions with 124k users and 24 million items. It is designed to test how well modern generative recommendation models handle continual learning, diversity, and multi-task objectives in a realistic dynamic environment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The 7-day sliding window and daily training schedule are not validated against real-world dynamics; the benchmark's claim to test continual learning under realistic distribution shift rests on this unvalidated temporal structure.","rationale":"The reader's weakest_assumption correctly identifies the 7-day sliding window and daily training schedule as the load-bearing assumption. The paper acknowledges this limitation but does not validate it. The concern is not that the assumption is false, but that it is unvalidated and central to the benchmark's contribution. The proposed concrete test would settle whether the temporal design is robust. The verdict should be CONDITIONAL because the paper's central claim depends on this assumption being reasonable; if the test shows sensitivity, the claim is weakened. The reader's verdict of ACCEPT is too strong given this unresolved concern. The paper is otherwise well-constructed and transparent, which is why the adjustment is to CONDITIONAL rather than REJECT.","tokens_in":33651,"tokens_out":559,"duration_ms":6878,"concrete_test":"Re-run the MiniOneRec continual learning experiment (Section 4.1) with two alternative temporal configurations: (a) a 14-day sliding window instead of 7 days, and (b) a twice-daily training schedule instead of daily. If the qualitative patterns (catastrophic forgetting, loss spikes at day boundaries, relative ordering of methods) change substantially, then the benchmark's temporal design is not robust and the central claim of realistic evaluation is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that KuaiRecV2 enables realistic evaluation of continual learning, diversity, and multi-task performance. The weakest load-bearing assumption is that the 7-day sliding window for item pools and the daily training schedule adequately capture the dynamics of real-world recommendation systems. The paper acknowledges this limitation in Appendix B: 'the data used in industrial recommendation systems can still be far more complex than ours... making it impossible for us to fully simulate real-world online recommendation systems.' However, the entire benchmark's claim to realism rests on this temporal structure. The paper does not provide any analysis showing that the 7-day window or daily schedule produces distribution shifts that are representative of real-world dynamics. Without validation, the observed catastrophic forgetting, diversity bottlenecks, and multi-task trade-offs could be artifacts of the specific temporal discretization rather than fundamental challenges of generative recommenders. The paper also does not compare against alternative window sizes or training schedules to demonstrate robustness. This is a load-bearing concern because if the temporal structure is not representative, the benchmark's primary contribution—narrowing the gap between offline and online evaluation—is undermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces KuaiRecV2, a large-scale benchmark dataset and evaluation pipeline for continual learning, diversity, and multi-task recommendation. The dataset comprises 124k users, 24M items, and 484M interactions over 30 days from Kuaishou, with multi-modal item features and semantic IDs. The benchmark includes a long-term recommendation pipeline with daily training, dynamic item pools (7-day sliding window), and augmented evaluation metrics covering retrieval accuracy, multi-level diversity (item and token level), and multi-task performance. The paper presents baseline results showing that generative recommenders outperform traditional ones, but suffer from catastrophic forgetting, diversity bottlenecks, and multi-task trade-offs. The central claim is that KuaiRecV2 provides a rigorous rubric where effective advancement requires improvement on all metrics.","tokens_in":33941,"tokens_out":2087,"duration_ms":17014,"significance":"If the benchmark is adopted, it could significantly advance the evaluation of recommender systems by incorporating continual learning, diversity, and multi-task objectives in a realistic temporal setting. The dataset size and temporal structure are substantial, and the inclusion of semantic IDs and multi-modal embeddings supports modern generative recommendation research. The paper's emphasis on a unified rubric (improvement on all metrics) is a strong contribution to preventing over-optimization on single metrics. The baseline experiments are thorough and reveal important challenges. The benchmark fills a gap in existing resources (FORGE, MUSE, KuaiRec) by focusing on continual learning and multi-task evaluation.","major_comments":[{"comment":"The 7-day sliding window for item pools and the daily training schedule are central to the benchmark's claim of realism for continual learning. However, the paper does not provide any validation that this temporal structure produces distribution shifts representative of real-world dynamics. The authors acknowledge in Appendix B that 'the data used in industrial recommendation systems can still be far more complex than ours... making it impossible for us to fully simulate real-world online recommendation systems.' While this is honest, the benchmark's primary contribution—narrowing the gap between offline and online evaluation—rests on the assumption that the 7-day window and daily schedule are meaningful proxies. The paper does not analyze alternative window sizes (e.g., 3-day, 14-day) or training frequencies (e.g., every 2 days) to demonstrate robustness. Without such analysis, the observed catastrophic forgetting, diversity bottlenecks, and multi-task trade-offs could be artifacts of the specific temporal discretization rather than fundamental challenges of generative recommenders. I recommend that the authors either provide a sensitivity analysis of the window size and training schedule, or explicitly reframe the benchmark's contribution as a controlled laboratory setting rather than a realistic simulation, and adjust the claims accordingly.","section":"§2.2, Figure 1, Appendix B","authors_acknowledge":"Our work provides the data and fundamental framework for continual recommendation. 25 B Limitations and Future Directions B.1 Limitations While KuaiRecV2 moves the field towards a more online-alike evaluation of recommender systems, the data used in industrial recommendation systems can still be far more complex than ours (e.g., excluding user features for privacy concerns), making it impossible for us to fully simulate real-world online recommendation systems.","authors_acknowledge_p":0.63},{"comment":"The backward transfer results in Table 3 show catastrophic forgetting, but the evaluation uses only HR@50 on Day 1. The paper does not report whether the forgetting is uniform across behaviors or item types. Given the multi-task and diversity focus, it would be informative to report backward transfer for other metrics (e.g., NDCG, diversity metrics) and for other days (e.g., Day-2 retention). Additionally, the replay baseline retains only 5% of previous instances; the paper does not discuss why 5% was chosen or whether results are sensitive to this hyperparameter. Without this, the claim that 'all methods show substantial forgetting' is supported but the analysis is shallow for a benchmark paper.","section":"§4.1, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The token-level diversity metrics are a novel contribution, but the paper does not explain how they are computed in sufficient detail for reproducibility. Appendix D provides brief definitions, but key implementation choices are missing: how are 'TopK Layer Diversity' and 'Average Path Diversity' exactly computed? Are they averaged over users? Over days? The paper states 'Detailed explanations are provided in the appendix D' but the appendix is only a few lines. For a benchmark, these metrics must be precisely defined. I recommend providing formal definitions with equations, and releasing the evaluation code to ensure reproducibility.","section":"§3.1, §4.2, Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The RL experiments compare SFT+RL alternating, RL-only, and SFT-only, but the RL-only strategy is applied starting from Day 20 without any SFT initialization. This is an unfair comparison because the RL-only model has not been trained on the data distribution before Day 20. The paper acknowledges that 'RL alone is insufficient to adapt to new data distributions,' but the experimental design conflates lack of SFT pretraining with inability to adapt. A fairer comparison would initialize the RL-only model with the Day 19 SFT checkpoint (same as the alternating strategy) and then apply only RL from Day 20 onward. The current setup overstates the limitation of RL. I recommend rerunning this experiment with proper initialization or clarifying that the RL-only condition is intentionally disadvantaged.","section":"§4.3, Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The scalability comparison on KuaiRecV2-Small shows that generative models outperform traditional models, but the traditional models (SASRec, GRURec) are trained with negative sampling (num_negatives=1000) while generative models use full vocabulary generation. This is not an apples-to-apples comparison: traditional models are evaluated on a sampled subset, which may underestimate their performance. The paper should discuss how negative sampling affects the comparison, or provide results with full softmax or other approximations for traditional models. Without this, the claim that 'architectural choice is more critical than model size alone' is weakened.","section":"§4.4, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states 'an effective advancement occurs if and only if the new method improves all metrics.' This is a strong criterion that may discourage research on specialized improvements. Consider softening to 'improves all metrics or provides a clear trade-off analysis.'","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'combinatorial diversity challenge' is introduced but not clearly defined until later. Consider providing a brief definition in the introduction.","section":"§1.1, RQ2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sparsity is reported as 99.9843% and 99.9871%. These values seem extremely high; please verify the calculation (e.g., sparsity = 1 - (#records)/(#users * #items)). With 124k users and 24M items, the total possible interactions is ~3 trillion, so 484M interactions gives sparsity ~99.984%, which is plausible but worth double-checking.","section":"§2.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The RL reward design is described in Appendix E.3, but the main text mentions 'reward functions, formatted as RB = Pb∈B αbRb' without defining αb. Please provide the α values used in experiments or state that they are tuned.","section":"§3.2.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The loss curve shows spikes at day boundaries, but the y-axis is loss and the inset shows initial loss. It would be helpful to also show validation metrics (e.g., HR@50) over days to correlate loss spikes with performance degradation.","section":"§4.1, Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Gini coefficient values are reported as percentages (×10−2). Please clarify whether these are Gini indices (0-1) scaled by 100, or something else. The values around 90% seem high; typical Gini for recommendation is lower. Provide interpretation.","section":"§4.2, Table 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The hyperparameter lists for SASRec and GRURec include 'item_num=2452987' which matches the Small dataset. Please confirm that the FullMonth experiments use the same item_num or adjust accordingly.","section":"Appendix E.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The tag frequency quantiles show median=1, meaning most tags appear once. This is an important characteristic but is not discussed in the main paper. Consider mentioning in §2.1 or §4.2.","section":"Appendix F.2, Table 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'KuaiRecV2' as the benchmark name, but the original KuaiRec [24] is a fully-observed dataset. The naming might cause confusion. Consider adding a brief note distinguishing the two.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Reference [50] and [51] are duplicates of [29] and [30] respectively. Please remove duplicates.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"KuaiRecV2: Benchmark exposes three hidden flaws in generative recommenders","keywords":["recommender systems","continual learning","generative recommendation","diversity","multi-task learning","benchmark","semantic IDs","catastrophic forgetting"],"falsifier":"If a generative recommender trained on KuaiRecV2’s 30-day stream maintains high retrieval accuracy on Day 1 test instances after training through Day 30 (backward transfer HR@50 > 30), while simultaneously achieving high token-level diversity (average layer unique tokens > 40 at layer 8) and positive multi-task metrics across all behaviors, then the claim that current methods fail under continual learning would be weakened.","tokens_in":36944,"feed_emoji":"📊","tokens_out":1668,"duration_ms":24585,"temperature":0.7,"pith_summary":"This paper introduces KuaiRecV2, a large-scale benchmark for evaluating generative sequential recommenders in a more realistic setting that combines continual learning, diversity, and multi-task objectives. The authors argue that existing offline evaluations miss critical failures—catastrophic forgetting, diversity collapse, and multi-task trade-offs—that appear when models face daily item-pool changes and shifting user interests over 30 days. They provide a dataset of 124k users, 24 million items, and 484 million interactions from Kuaishou, with multi-modal item features and a dynamic candidate pool that refreshes every seven days. The central claim is that a method advances the state of the art only if it improves retrieval accuracy, diversity, and multi-task metrics simultaneously; anything less is a ‘defective advancement.’ Experiments show that even state-of-the-art generative models suffer from sharp performance drops when item pools change, that token-level diversity is concentrated in the first few tokens, and that reinforcement learning alone fails under distribution shift unless combined with supervised fine-tuning. If the benchmark’s temporal structure captures real-world dynamics, then many published gains on static benchmarks may not carry over to production settings, and the field needs new algorithms that explicitly balance adaptation, retention, diversity, and multiple user behaviors.","feed_headline":"KuaiRecV2: Benchmark exposes three hidden flaws in generative recommenders","feed_subtitle":"A 30-day streaming dataset shows that state-of-the-art models forget, lack diversity, and fail under distribution shift unless trained…","key_machinery":"The key machinery is the long-term recommendation pipeline with a 7-day sliding window for the item candidate pool. Each day, items created more than seven days ago are removed and newly created items are added; models are trained sequentially on each day’s data, initialized from the previous day’s checkpoint, and evaluated daily on a leave-one-out task. This temporal structure forces models to confront distribution shifts, cold-start items, and forgetting, and it grounds the benchmark’s three evaluation axes: retrieval accuracy, token-level and item-level diversity, and multi-task performance across behaviors (click, long view, like, follow, forward, comment, search, profile enter, hate).","core_discovery":"The paper’s central discovery is that generative sequential recommenders, when evaluated under a continual-learning rubric with daily item-pool refreshes, exhibit three systematic weaknesses that are invisible in static benchmarks: (1) catastrophic forgetting of earlier user preferences as new items enter the pool, with back-transfer HR@50 dropping from 39 to 0.19 over seven days; (2) a combinatorial diversity bottleneck in which the first few tokens of a semantic ID determine most of the diversity while later tokens are nearly deterministic; and (3) a fragility of reinforcement-learning-based multi-task alignment, which improves accuracy on the current day but collapses when the item distribution shifts unless RL is interleaved with supervised fine-tuning. The authors operationalize an ‘effective advancement’ as a method that improves retrieval accuracy, multi-level diversity, and multi-task metrics all at once, and they show that no current method meets this bar on KuaiRecV2.","pith_inferences":["The 7-day sliding window is a plausible but untested proxy for real-world item churn; an experiment that varies the window length (e.g., 1 day, 14 days, 30 days) would reveal how sensitive the observed forgetting and diversity bottlenecks are to this temporal assumption.","The benchmark’s exclusion of user profile features for privacy means that methods relying on user-side embeddings are under-served; a synthetic user-feature variant could test whether additional user context helps mitigate forgetting.","The SID collision rate (0.81%) is low, but the tag distribution is extremely long-tailed (median frequency 1); this suggests that semantic diversity at the fine-grained level is sparse, and token-level diversity metrics may conflate semantic sparsity with model diversity failure.","The paper does not report confidence intervals or statistical significance tests; the reported standard deviations across three seeds are small, but without significance tests it is unclear whether observed differences between models are reliable."],"forward_implications":["Generative recommenders that rely on semantic IDs (SIDs) will need explicit mechanisms to retain knowledge of earlier items when the pool turns over, because even replay buffers and elastic weight consolidation only partially mitigate the observed forgetting.","Token-level diversity metrics reveal that diversity control must operate on the first few tokens of an SID; later tokens contribute little variety, so beam-search strategies that treat all tokens equally will fail to produce diverse recommendations.","Multi-task reinforcement learning for recommender systems should be combined with supervised fine-tuning on the current day’s data rather than applied alone, because RL-only training cannot adapt to item-pool changes.","The benchmark’s ‘effective advancement’ criterion—improving accuracy, diversity, and multi-task metrics simultaneously—raises the bar for future work and may disqualify methods that trade one objective for another."],"supporting_citations":[{"why":"Introduces the TIGER model and the concept of learned Semantic IDs (SIDs) for generative retrieval, which KuaiRecV2 adopts as item representations.","marker":"[8]"},{"why":"Describes the OneRec system that combines SFT and RL for generative recommendation; KuaiRecV2 uses a similar hybrid training strategy and compares against it.","marker":"[10]"},{"why":"KuaiRec is the predecessor dataset; the paper positions KuaiRecV2 as a much larger, dynamic, multi-task extension of that earlier fully-observed benchmark.","marker":"[24]"},{"why":"FORGE is a related benchmark for SID quality assessment; the paper contrasts KuaiRecV2’s focus on continual learning and diversity with FORGE’s focus on SID quality.","marker":"[25]"},{"why":"SASRec is a standard sequential recommendation baseline; the paper implements it as one of the discriminative baselines to compare against generative models.","marker":"[44]"},{"why":"TIGER is a generative sequential model that the paper reimplements as a baseline; it uses the same SID tokenization that KuaiRecV2 provides.","marker":"[46]"},{"why":"MiniOneRec is the primary generative model used in the paper’s experiments; it is a lightweight open-source generative recommender based on Qwen2.5.","marker":"[48]"},{"why":"MUSE is a benchmark for lifelong user interest modeling; the paper contrasts KuaiRecV2’s focus on continual learning and diversity with MUSE’s focus on user interest modeling.","marker":"[49]"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The 7-day sliding window for item pools and the daily training schedule are assumed to capture the essential dynamics of real-world recommendation systems, but the authors acknowledge that industrial systems are far more complex, making it impossible to fully simulate online conditions.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"cost":0.000476021,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000476021,"upstream_inference_prompt_cost":0.000102011,"upstream_inference_completions_cost":0.00037401},"total_tokens":3176,"prompt_tokens":1121,"completion_tokens":2055,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1121,"tokens_out":2055,"cost_source":"openrouter_usage","duration_ms":12803,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T12:32:41.549496+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a generative recommender trained on KuaiRecV2’s 30-day stream maintains high retrieval accuracy on Day 1 test instances after training through Day 30 (backward transfer HR@50 > 30), while simultaneously achieving high token-level diversity (average layer unique tokens > 40 at layer 8) and positive multi-task metrics across all behaviors, then the claim that current methods fail under continual learning would be weakened.","supporting_citations":[{"cited_title":"Onerec technical report.CoRR, abs/2506.13695, 2025","cited_arxiv_id":null,"evidence_quote":"Describes the OneRec system that combines SFT and RL for generative recommendation; KuaiRecV2 uses a similar hybrid training strategy and compares against it."},{"cited_title":"Kuairec: A fully-observed dataset and insights for evaluating recommender systems","cited_arxiv_id":null,"evidence_quote":"KuaiRec is the predecessor dataset; the paper positions KuaiRecV2 as a much larger, dynamic, multi-task extension of that earlier fully-observed benchmark."},{"cited_title":"FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets","cited_arxiv_id":"2509.20904","evidence_quote":"FORGE is a related benchmark for SID quality assessment; the paper contrasts KuaiRecV2’s focus on continual learning and diversity with FORGE’s focus on SID quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SASRec is a standard sequential recommendation baseline; the paper implements it as one of the discriminative baselines to compare against generative models."},{"cited_title":"Tran, Jonah Samost, Maciej Kula, Ed H","cited_arxiv_id":null,"evidence_quote":"TIGER is a generative sequential model that the paper reimplements as a baseline; it uses the same SID tokenization that KuaiRecV2 provides."},{"cited_title":"MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation","cited_arxiv_id":null,"evidence_quote":"MiniOneRec is the primary generative model used in the paper’s experiments; it is a lightweight open-source generative recommender based on Qwen2.5."},{"cited_title":"MUSE: A simple yet effective multimodal search-based framework for lifelong user interest modeling.CoRR, abs/2512.07216, 2025","cited_arxiv_id":null,"evidence_quote":"MUSE is a benchmark for lifelong user interest modeling; the paper contrasts KuaiRecV2’s focus on continual learning and diversity with MUSE’s focus on user interest modeling."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}