{"id":"5c4e006c-b6c3-4c3e-9c60-737cecbdcea7","arxiv_id":"2411.10057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"KuaiFormer is a Transformer-based retrieval model with multi-interest query tokens and adaptive sequence compression, deployed at Kuaishou with reported online watch-time gains of 0.126% to 0.411%.","lead":"Kuaishou deployed KuaiFormer, a Transformer-based retrieval model for short-video recommendations, and reports small but meaningful gains in user watch time. The paper explains three design choices that made a large-scale Transformer retrieval model practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing concern is attribution: the online watch-time gains in Table 2 may reflect the added KuaiFormer retrieval pathway and extra candidate supply, not the Transformer architecture itself, because no control for candidate count or ranking is reported.","rationale":"The paper is a credible engineering report: the architecture is well specified, the logQ-corrected in-batch softmax with label smoothing is a standard and appropriate training objective for billion-scale items, the compression scheme is clearly described with complexity numbers, and the offline replay comparison in Table 1 gives KuaiFormer an edge over strong baselines. I am not claiming the online effects are fraudulent or that the system is not deployed. The specific underdetermination I flag is methodological: adding a retrieval pathway changes the candidate union, and without a control for that added supply the causal attribution of the A/B lift to the Transformer/multi-interest design is not established. The reader's weaker-assumption point about offline Accuracy being training accuracy is valid but secondary: it affects the hyperparameter and compression ablations (RQ3), not the headline offline hit-rate comparison or the online claim. My recommendation remains CONDITIONAL, so the reader's verdict should not change: accept with the request that the authors release the A/B specification, confidence intervals, and an equal-candidate-count control, or clearly narrow the claim to 'adding KuaiFormer as an extra pathway improves watch time.'","tokens_in":12579,"tokens_out":8510,"duration_ms":100250,"concrete_test":"Obtain the A/B cohort-level daily watch-time logs and experiment specification for the three scenarios, and compute the treatment effect with confidence intervals after controlling for total candidate count and whether KuaiFormer items were already available via other pathways. The decisive version is to run a control arm that adds an equal-sized set of random or baseline retrieval candidates in place of KuaiFormer before aggregation; if the watch-time lift is not robust to that substitution, or if the confidence intervals include zero, the gains cannot be attributed to the KuaiFormer architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 says KuaiFormer was introduced as a new retrieval pathway whose candidates are aggregated and deduplicated with all other pathways before ranking. Table 2 reports one-week A/B gains of +0.360%/+0.126%/+0.411% video watch time on 10% of users, but gives no confidence intervals, significance tests, or sample sizes, and does not state whether the control condition compensated for the extra candidate supply (e.g., by adding an equal-sized random or baseline pathway) or held the downstream ranking fixed. If the treatment simply enlarges the candidate union, even a mediocre retrieval path can raise coverage and produce positive watch-time movement; the architecture-specific claims (pure-Transformer, multi-interest, compression) are then not the demonstrated cause. The offline 'online Hit Rate' replay in Table 1 checks only whether a viewed item falls in each model's top-K; it does not measure marginal contribution to the final ranking or watch time. This makes the central headline result underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KuaiFormer, a Transformer-based retrieval model for Kuaishou's short-video recommendation system. The architecture replaces the conventional dual-tower EBR score-estimation setup with a Next Action Prediction paradigm: an MLP/embedding layer produces token representations of watched videos, a Llama-style causal Transformer processes a compressed sequence, and multiple learnable query tokens produce several interest representations. The training objective combines in-batch softmax, logQ sampling correction, and label smoothing. The authors claim deployment since May 2024, serving over 400 million daily active users, with offline hit-rate improvements over six baselines and online A/B gains of +0.360%, +0.126%, and +0.411% in video watch time in the three largest scenarios. The paper also presents hyperparameter studies for sequence length, number of query tokens, number of layers, and item compression.","tokens_in":12738,"tokens_out":4070,"duration_ms":47866,"significance":"If the central claims were fully evidenced, this would be a valuable industrial case study: it demonstrates a pure Transformer retrieval model at billion-scale candidate-pool size, with a plausible treatment of multi-interest extraction and long-sequence efficiency. The detailed architecture description and deployment pipeline are useful to practitioners. However, the evaluation evidence as presented is not sufficient to establish the architectural attribution: the offline replay table lacks interval estimates, the hyperparameter analysis relies on training-set accuracy, and the online A/B section does not control for candidate-supply effects. The paper's stated contribution ('first real-time retrieval model by pure Transformer in industrial-scale RecSys') is a strong claim that needs stronger evidence.","major_comments":[{"comment":"The Offline Accuracy metric is defined as the accuracy when training loss stabilizes, which measures fit to the training behavior stream rather than held-out generalization. RQ3 and the claims about sequence length, query tokens, layer count, and the assertion that compression 'reduces noise' are based entirely on this metric (Figure 3 and Table 3). Training accuracy can rise with model capacity or sequence length without implying better retrieval on unseen requests, especially under in-batch softmax training. I request an evaluation on a temporally disjoint held-out log (e.g., next-item prediction with top-K recall) before these architectural conclusions are treated as established.","section":"§4.1.2, §4.4, Table 3"},{"comment":"Table 1 reports only point estimates of HR@50 through HR@1000 for KuaiFormer and the six baselines, with no confidence intervals, number of requests, or significance tests. The headline comparisons (e.g., 26.30% relative improvement over the runner-up at HR@1000) could be within noise. The paper should report interval estimates across users or days and significance tests for the difference with the runner-up.","section":"Table 1, §4.2"},{"comment":"The online A/B section gives only relative percentage lifts and no p-values, confidence intervals, sample sizes, or variance measures, despite the statement that a ~0.1% video-watch-time change is 'statistically significant'. Moreover, because §3 states that KuaiFormer was added as a new retrieval pathway and candidates from all pathways are aggregated and deduplicated before ranking, the reported gains may reflect the added candidate supply rather than the Transformer architecture itself. The treatment should be compared against a control with an equal-sized additional baseline pathway or with the total candidate count and downstream ranking held fixed.","section":"§4.3, Table 2"},{"comment":"Equation (10) defines Score_{x_{n+1}} as argmax({x^T u_1, ..., x^T u_k}); argmax returns an index rather than a score value and is non-differentiable, which is inconsistent with using Score in the in-batch softmax loss in Eq. (11). If the intended operation is max or a differentiable approximation, this should be stated explicitly; as written, the training objective is not well defined.","section":"Eq. (10), Eq. (11)"}],"minor_comments":[{"comment":"There are several typos and formatting issues, including 'recomenndation' in §3, 'thelargest' in §4.1.1, and broken equation/table rendering in Eq. (10) and Table 3. A careful copyedit is needed.","section":"Throughout"},{"comment":"The compression split is not stated precisely: the text says the sequence is divided into earlier, middle, and latest parts, and Eq. (7) uses group sizes 64 and 16, but the total of the displayed groups (128 + 80 = 208) and the number of uncompressed items (48) should be tied explicitly to the final compressed length of 55 in Figure 1.","section":"§2.3, Eq. (7)"},{"comment":"The bucketing formula for duration uses a 1000-bucket uniform strategy with maximum 300s, but the paper does not state how other continuous attributes are bucketed or whether each attribute has its own bucket count. Clarifying this would improve reproducibility.","section":"§2.2.1, Eq. (4)"},{"comment":"The label smoothing loss in Eq. (12) introduces alpha but the paper does not state its chosen value or sensitivity to it; since the abstract and method sections emphasize stable training, a short ablation or at least a stated default would be helpful.","section":"§2.5, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an industrial practice report with potentially useful deployment insights, but the evidence as presented does not support the headline attribution claims. The main fixes needed are statistical rigor (confidence intervals and tests), a control for candidate supply in the A/B comparison, and held-out evaluation for the hyperparameter analysis. If the authors provide these, a revised version could be acceptable for an industrial-practice venue. I see no sign of fabrication, but the very broad 'first' claim should be softened unless the authors verify it against a known literature landscape."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you want to see how a pure Transformer retrieval model can be made to work at Kuaishou scale. The paper is a genuine engineering contribution, but the evidence for the architecture-specific benefits is weaker than the headline claims.\n\nWhat's new: the specific combination of a causal Transformer with multiple learnable query tokens that attend to compressed item groups, plus the logQ-corrected smoothed in-batch softmax. The deployment details are genuinely useful: adaptive compression cuts compute substantially (a 256-length sequence with only 10% more cost than 64), and the minute-level online update is a real achievement. The paper also gives a sensible treatment of the challenges of in-batch softmax with billion-item corpora.\n\nThe soft spots are in the evaluation. The offline 'Accuracy' in Section 4.1.2 is training accuracy when loss stabilizes--that measures fit to training data, not generalization. Section 4.4 then uses this metric to conclude that compression reduces noise, which is not supported. The online A/B in Table 2 reports watch-time gains of +0.360%, +0.126%, and +0.411% with no confidence intervals, no significance tests, and no sample sizes. The assertion that 'about 0.1% is statistically significant' is just asserted, not demonstrated. The deployment section says KuaiFormer is added as a new retrieval pathway with candidates aggregated and deduplicated with all others. There is no control for candidate count or for downstream ranking. So the gains could come from the extra candidate supply, not from the Transformer architecture per se. The 'first real-time pure Transformer retrieval' claim is also stated without a thorough comparison to PinnerFormer or other industrial work.\n\nThat said, these flaws don't sink the engineering value. The architecture is coherent and the deployment experience is worth having. But the claims in the abstract and conclusion go beyond what the evidence shows.\n\nI would send this to peer review any day--it's exactly the kind of paper that gives the RecSys community a real data point on industrial Transformer retrieval. But I would ask for a revision that adds a held-out evaluation, reports uncertainty on the online metrics, and addresses the attribution problem--either with a candidate-count control or a clear statement of why the comparison is still meaningful. If those are not possible, the claims should be toned down to match the evidence.","headline":"A credible industrial Transformer-retrieval deployment with a clear architecture, but the empirical claims are underdetermined: no held-out evaluation, no significance tests, and no control for the extra candidate supply in the online A/B.","tokens_in":13329,"tokens_out":2013,"would_cite":true,"duration_ms":21396,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a pure-Transformer retrieval model, KuaiFormer, lifted short-video watch time in Kuaishou's three largest scenarios after going live in May 2024.","keywords":["short-video recommendation","transformer","user interest modeling","multi-interest query tokens","adaptive item compression","next action prediction","in-batch softmax","industrial retrieval"],"falsifier":"Re-run the Section 4.4 hyperparameter comparisons (sequence length, query-token count, layer depth, compression strategy) using held-out next-item prediction or online hit rate on logs from a different day; if the compressed 256-item model does not beat the uncompressed 256-item model on held-out data, the claim that compression reduces noise rather than merely improving training fit is refuted.","tokens_in":12326,"feed_emoji":"🎬","tokens_out":9291,"duration_ms":82275,"temperature":0.7,"pith_summary":"KuaiFormer is a retrieval-stage model for short-video recommendation that replaces the usual score-estimation task (predicting click-through rate) with a transformer-driven Next Action Prediction task: given a user's recently watched videos, it predicts which video the user will engage with next. The paper argues that this shift, together with multiple learnable query tokens that extract several distinct interest vectors and an adaptive compression mechanism that coarsens older watched items, lets a pure Transformer architecture work at industrial scale for the first time in retrieval. The model has been live in the Kuaishou app since May 2024, serving over 400 million daily active users, and online A/B tests in the three largest scenarios report video watch time gains of +0.360%, +0.126%, and +0.411%. A sympathetic reader would care because the paper claims to close the gap between academic Transformer recommenders and the efficiency and scale constraints of real deployments, where sequence length and billion-item candidate pools usually force simpler dual-tower models.","feed_headline":"Pure-Transformer retrieval lifts Kuaishou watch time by 0.411%","feed_subtitle":"KuaiFormer runs live for over 400 million daily users and beats deployed retrieval baselines on offline hit rate.","key_machinery":"The load-bearing machinery is an autoregressive Transformer backbone (Llama-style, with RMSNorm, masked multi-head self-attention, and a pointwise feed-forward layer) that consumes item tokens and, appended after them, a set of learnable query tokens under the same causal mask, so each successive interest vector can attend to earlier interest vectors and to the whole item sequence. An adaptive compression stage splits the behavior history into early, middle, and latest parts, compresses groups of 64 and 16 old items via a single-layer bidirectional Transformer followed by mean pooling, and concatenates the compressed tokens with the latest uncompressed items; this turns a 256-item history into 55 tokens and is what keeps quadratic self-attention affordable. The training objective is a smoothed in-batch softmax with logQ correction and label smoothing, which stabilizes learning over a billion-scale candidate set and tolerates noisy short-video labels. Together these mechanisms carry the paper's three claims: multi-interest extraction, long-sequence efficiency, and stable billion-item training.","core_discovery":"The central claim is that retrieval can be redefined as Next Action Prediction with a pure Transformer encoder instead of a score-estimation task such as click-through rate prediction. Given the user's recent watched-video sequence, KuaiFormer produces several user interest vectors from learnable query tokens; the score for a candidate video is the maximum inner product between the video embedding and those interest vectors. To make this tractable, the paper introduces adaptive item compression, which groups and mean-pools older items through a one-layer bidirectional Transformer, and a smooth in-batch softmax loss with logQ correction, which avoids computing over the full billion-item vocabulary. In offline replay of real requests, KuaiFormer reports the highest hit rate among deployed baselines (for example, HR@1000 of 19.88% versus 10.26% for the GNN pathway), and the paper states that this is the first real-time retrieval model built on a pure Transformer architecture at industrial scale.","pith_inferences":["The causal masking between query tokens is a testable design choice: an ablation with independent, non-causal query tokens would isolate whether the reported gains come from interest disentanglement through autoregressive interaction or merely from having several output vectors.","The compression schedule (64/16 grouping with the latest 48 items uncompressed) is reported for one configuration; whether it generalizes to other session lengths or video categories is open, since the memory-decay assumption it rests on is not directly measured.","One could make the number of query tokens and the compression window adaptive to each user's recent activity, since the paper fixes both as global hyperparameters; diversity of recent views would be a natural signal for choosing how many interest vectors to output.","The same Next Action Prediction recipe with multiple query tokens and compressed histories should transfer to other feed-style products such as e-commerce or long-video recommendation, where the bottlenecks are the same: billion-scale item sets and multiple simultaneous interests."],"forward_implications":["Retrieval no longer needs a dual-tower interaction at the top layer; a single Transformer can produce user vectors that are compared with item embeddings by inner product, so the same ANN infrastructure used for embedding retrieval remains usable.","Longer behavior histories become affordable: compressing 256 items into 55 tokens costs roughly 10% extra compute over a 64-item baseline, so industrial systems can extend context without paying the full quadratic attention price.","Multi-interest is expressed as several output vectors rather than one, which increases retrieval coverage and, according to the paper, improves the 'Novel Surprise' metric that measures discovery of new interests.","The Next Action Prediction objective with smoothed in-batch softmax provides a recipe for training Transformer recommenders on billion-scale item sets without full softmax over the candidate pool.","Deployment is feasible as an additional retrieval pathway alongside existing item2item, graph, and multi-interest pathways, with minute-level online updates and GPU brute-force TopK serving."],"supporting_citations":[{"why":"Supplies the Transformer backbone, multi-head self-attention, that KuaiFormer adapts for retrieval.","marker":"[32]"},{"why":"Provides the SASRec autoregressive sequential baseline KuaiFormer extends and compares against.","marker":"[14]"},{"why":"ComiRec is the deployed multi-interest baseline whose dynamic routing and self-attention interest extraction KuaiFormer surpasses.","marker":"[4]"},{"why":"MIND supplies the capsule-network multi-interest idea that motivates multiple interest vectors.","marker":"[18]"},{"why":"Provides the logQ correction used to remove in-batch softmax sampling bias.","marker":"[35]"},{"why":"Provides label smoothing, used to soften positive/negative labels for noisy short-video watching signals.","marker":"[24]"},{"why":"Defines the Llama-style backbone (RMSNorm, masked self-attention, pointwise feedforward) KuaiFormer adopts.","marker":"[31]"},{"why":"Faiss is the ANN search tool used for TopK candidate retrieval at serving time.","marker":"[13]"},{"why":"PinnerFormer is the daily-updated industrial Transformer baseline that motivates KuaiFormer's real-time adaptation.","marker":"[25]"}],"fun_headline_variants":["Pure Transformer retrieval cuts loss with Next Action Prediction","KuaiFormer: Next Action Prediction for billion-scale retrieval","Transformer retrieval lifts watch time by 0.411% at 400M DAU","Retrieval redefined: Next Action Prediction outperforms CTR","From score estimation to next action: transformer retrieval at scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's offline evidence for its design choices rests on 'Accuracy' measured as the model's fit to its own training data once the loss stops falling, not on held-out user behavior; if that training fit does not track retrieval quality on unseen logs, the claimed benefits of query tokens and compression are not demonstrated by the offline experiments.","fun_headline_variants_meta":{"raw":{"variants":["Pure Transformer retrieval cuts loss with Next Action Prediction","KuaiFormer: Next Action Prediction for billion-scale retrieval","Transformer retrieval lifts watch time by 0.411% at 400M DAU","Retrieval redefined: Next Action Prediction outperforms CTR","From score estimation to next action: transformer retrieval at scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1756,"prompt_tokens":982,"completion_tokens":774,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":688}},"tokens_in":598,"tokens_out":774,"duration_ms":8652,"temperature":1.0,"reasoning_tokens":688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:00:52.345066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Section 4.4 hyperparameter comparisons (sequence length, query-token count, layer depth, compression strategy) using held-out next-item prediction or online hit rate on logs from a different day; if the compressed 256-item model does not beat the uncompressed 256-item model on held-out data, the claim that compression reduces noise rather than merely improving training fit is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer backbone, multi-head self-attention, that KuaiFormer adapts for retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SASRec autoregressive sequential baseline KuaiFormer extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MIND supplies the capsule-network multi-interest idea that motivates multiple interest vectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the logQ correction used to remove in-batch softmax sampling bias."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides label smoothing, used to soften positive/negative labels for noisy short-video watching signals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Llama-style backbone (RMSNorm, masked self-attention, pointwise feedforward) KuaiFormer adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PinnerFormer is the daily-updated industrial Transformer baseline that motivates KuaiFormer's real-time adaptation."}],"review_version":1}