{"id":"4b64c2fa-c964-4d75-a48c-f5537165b094","arxiv_id":"2508.10615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FuXi-β shows that removing query-key attention and using a functional relative time bias makes generative recommendation Transformers faster and, on industrial datasets, more accurate.","lead":"A new recommendation model, FuXi-β, removes the query-key attention map and replaces expensive bucketed time-bias with a simple learnable power function, making Transformer recommenders faster to train while matching or improving accuracy. On private industrial datasets the authors report 27-47% higher NDCG@10 than FuXi-α with 16-21% less training time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Industrial 27–47% NDCG@10 gain is not yet attributable to the architecture: fixed-hyperparameter comparison, no industrial ablations, and anomalous metric pattern leave tuning/evaluation artifacts as a competing explanation.","rationale":"The most load-bearing assumption in the paper is that the industrial improvement is caused by the architecture rather than by the comparison setup. The reader's weakest_assumption captures this, and I agree. The paper's own public-dataset experiments show at best parity (Table 3), so the large industrial gains are unexplained by the stated mechanism. The overfitting explanation is plausible but is supported only by the absence of evidence, not by training curves or industrial ablations. The fixed-hyperparameter protocol could easily disadvantage FuXi-α, especially if it has different optimal regularization or learning-rate schedules; without a tuning report, this competing explanation is live. The metric pattern on Daily Recommendations (HR@50 nearly flat while NDCG@10 rises 47%) further suggests the gain may not be a broad ranking improvement but a top-of-ranking/calibration effect. A controlled ablation and a tuned-baseline rerun would settle the point. If the gap survives, the CONDITIONAL verdict can be upgraded; if not, the headline claim should be retracted. Secondary issues, such as the dimensional inconsistency in Eq. (14), should be corrected but are less decisive because the public code can disambiguate the intended implementation.","tokens_in":19768,"tokens_out":10016,"duration_ms":105900,"concrete_test":"On the two industrial datasets, rerun Table 4 with the FuXi-α baseline selected by a small hyperparameter sweep (e.g., 3 learning rates × 2 dropouts) rather than the fixed setting, while keeping all data splits and training budgets identical. If the best tuned FuXi-α's NDCG@10 rises substantially toward FuXi-β (or surpasses it), the reported 27–47% advantage is an artifact of under-tuned baselines, not the architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is a 27–47% relative improvement in NDCG@10 over FuXi-α on two private industrial datasets (Table 4). The stated mechanism is that removing the query-key attention map and replacing bucketed temporal bias with FRAB reduces overfitting (Sec. 6.2.2). This attribution is not supported by the reported evidence. (1) All ablations of the two components (Tables 5–7) are on MovieLens datasets; no industrial ablation isolates the contribution of query-key removal or FRAB. On ML-1M, removing the query-key map alone degrades NDCG@10 (0.1797 vs 0.1835), and on ML-20M the gain is negligible; the combined model is near parity on both public sets. There is no industrial evidence that either modification is responsible for the large gain. (2) 'Identical hyperparameters' (Sec. 6.1.4: embedding 256, layers 4 for all models) is not sufficient for a fair comparison; if FuXi-α has different optimal learning rate, dropout, or training schedule on these datasets, the gap could be a tuning artifact. The paper does not report the tuning protocol for industrial baselines. (3) The metric pattern in Table 4 is anomalous: on Daily Recommendations, HR@50 is nearly identical to FuXi-α (0.6497 vs 0.6449) while NDCG@10 jumps 47%, suggesting the improvement is concentrated at the very top of the ranking and may reflect differences in output calibration or a few head items rather than a broad ranking improvement. Without industrial training curves, an overfitting explanation is post hoc. If the comparison is unfair or the gain is not from the architecture, the headline claim—the basis for the paper's central conclusion—falls.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a lightweight framework for Transformer-like generative recommender models, with two main modifications: (1) Functional Relative Attention Bias (FRAB), replacing bucketed relative temporal attention bias with a learnable power function f(x)=a(1+x)^{-b}, and (2) an Attention-Free Token Mixer (AFTM) that removes the query-key attention map and relies on relative-position and relative-time bias maps as attention. The framework is instantiated as FuXi-β by modifying FuXi-α. Experiments on MovieLens-1M, MovieLens-20M, and two private industrial datasets report public-dataset parity with FuXi-α, training-time reductions, scaling-law behavior, and 27–47% NDCG@10 improvements over FuXi-α on the industrial datasets. The authors attribute the industrial gains primarily to reduced overfitting from removing the query-key attention map.","tokens_in":20246,"tokens_out":3736,"duration_ms":44857,"significance":"If the central claims hold, the paper provides a useful simplification result: query-key attention may be unnecessary in generative recommendation, and a parametric temporal bias can replace bucketed lookup with lower cost. The strengths include public code, a concrete complexity analysis (Table 1), compatibility experiments on LLaMa and HSTU, and scaling experiments on industrial data. The main significance, however, rests on the industrial NDCG gains and the causal attribution to architecture; as reported, that attribution is not directly supported by the experiments. The public-dataset ablations are mixed, the industrial datasets are private, and the baseline tuning protocol is not described. The central architectural claim is plausible but needs stronger evidence before the 27–47% improvement can be accepted as architecture-driven.","major_comments":[{"comment":"Equation (14) is dimensionally inconsistent as written. W_u and W_v are defined in R^{d x 2d}, so U = φ(XW_u) and V = φ(XW_v) are n x 2d. Then concat(BV, B_t V) is n x 4d, and the elementwise product U ⊙ concat(...) is invalid. The AFTM formulation must be corrected, e.g., by projecting to separate d-dimensional gates/values or by splitting U into two components. As written, the central module cannot be implemented reproducibly.","section":"Section 5.3, Eq. (14)"},{"comment":"The causal mask is not specified for the bias-only attention maps. In standard self-attention the mask is applied before normalization; here B and B_t serve directly as attention maps, but the paper does not state that their upper-triangular entries are masked or otherwise constrained. Without a causal mask, future items leak into earlier positions, which would break the autoregressive training objective defined in Section 5.5. If masking is applied in code, it must be stated explicitly in Eq. (14) or the surrounding text.","section":"Section 5.3 / Eq. (14)"},{"comment":"The 27–47% NDCG@10 improvements on industrial datasets are attributed to 'reduced overfitting' from removing the query-key map, but no industrial ablation or training/validation curve is provided. The public ablations do not consistently support the mechanism: Table 5 shows that removing the query-key map alone degrades NDCG@10 on ML-1M (0.1797 vs. 0.1835), and the full FuXi-β is only at parity on both public datasets. The metric pattern in Table 4 is also anomalous: on Daily Recommendations, HR@50 changes by only 0.7% (0.6449 to 0.6497) while NDCG@10 changes by 47%, suggesting a top-rank redistribution that is not explained. Without industrial ablations or curves, the mechanism remains a post-hoc explanation.","section":"Section 6.2.2, Table 4"},{"comment":"The paper states that identical hyperparameters (embedding size 256, 4 layers) were used for all models on industrial data, but this does not establish a fair comparison. If FuXi-α, HSTU, or other baselines have different optimal learning rates, dropout, or schedules on these datasets, the gap could be a tuning artifact. The tuning protocol, search space, and final hyperparameters for baselines must be reported before the headline industrial superiority claim can be accepted.","section":"Section 6.1.4 / Section 6.2.2"}],"minor_comments":[{"comment":"The text says public performance results are presented in Table 2, but Table 2 is dataset statistics; the performance numbers are in Table 3. Similarly, Section 6.3.1 refers to 'Table 3 presents total training time', but Table 3 is the performance table. Please renumber tables or fix cross-references.","section":"Section 6.2.1 / Table 3"},{"comment":"f(x) = a(1+x)^{-b} is monotonically decreasing only when b > 0. The text calls it 'a monotonically function' without specifying the parameter sign. Clarify the parameterization or the intended monotonicity constraint.","section":"Eq. (12)"},{"comment":"In Table 6, the 'bucket' row is compared against 'pow' on ML-1M/ML-20M. It would help to specify whether 'bucket' uses the same number of buckets and the same bucketing function as HSTU/FuXi-α; otherwise the comparison is ambiguous.","section":"Section 6.4.2 / Table 6"},{"comment":"Panel (c) shows attention weight magnitudes up to 400 on ML-20M for 'linear' and 'mixed', while other variants are near zero. Please explain the scale or whether this is due to the learned parameters in those functions, since it affects the qualitative conclusions about suitable functions.","section":"Figure 3"},{"comment":"The terms 'query-key attention map', 'positional attention map', and 'temporal attention map' should be defined in the caption or text. In particular, the 'positional' and 'temporal' maps correspond to B and B_t in Eq. (14), and their precise construction under FRAB should be stated.","section":"Section 6.4.3 / Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's public code and complexity analysis are assets. The main concern is that the headline industrial result is not yet attributable to the architecture: the AFTM equation has a dimensional error, the causal mask is unspecified, and no industrial ablations or tuning details are given. These are fixable in revision, but the evidentiary bar for the industrial claim should be raised substantially. The authors should either add industrial ablations and baseline tuning reports or temper the causal language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine engineering contribution with a provocative empirical claim, but the industrial headline numbers are not yet backed by the evidence on the page. What's genuinely new here: the FRAB functional temporal bias, the attention-free token mixer that drops query-key attention, and the empirical observation that the query-key attention map can hurt in recommendation. The public-dataset speedups are real and consistent, around 5–27% training time reduction with parity in accuracy. The compatibility analysis across LLaMa and HSTU is honest, including the fact that HSTU-beta degrades. The code is public and the ablation tables are reported with full numbers, which is to their credit.\n\nThe soft spots are real, though some are minor. Equation (14) is dimensionally inconsistent as written: U and V are n×2d after projection, but concat(BV, B_t V) is n×4d, so the elementwise product doesn't type-check. That's likely a typo in the projection dims, but it needs fixing. The causal mask is never specified for the bias-only attention maps, and since those maps are the attention mechanism, masking is not optional.\n\nMore serious is the mixed support for the \"query-key maps are harmful\" claim. In Table 5, removing query-key on ML-1M hurts (0.1797 vs 0.1835) and on ML-20M is flat. In Table 7, removing query-key from the Large model helps on both datasets. So the paper's own ablations do not consistently support the headline mechanism. The function comparison in Table 6 is useful, and the power function choice is defensible.\n\nThe industrial 27–47% NDCG@10 gains are the load-bearing claim, and they are not yet supported. There are no industrial ablations isolating FRAB or query-key removal, and identical hyperparameters across all baselines is not a substitute for tuning evidence. The metric pattern on Daily Recommendations is anomalous: HR@50 is nearly flat (0.6497 vs 0.6449) while NDCG@10 jumps 47%. That could be a genuine concentration of improvement at the very top, but it could equally be calibration or evaluation artifact. The overfitting explanation is plausible but post hoc.\n\nThe \"scaling law\" claim is also overstated—they show monotonic improvement with depth up to 32 layers, which is not a scaling law in the usual sense.\n\nWho should read this: people building generative recommenders, especially industrial practitioners who care about training cost. It is worth a serious referee but not in current form. The fixes are manageable: correct Eq. 14, specify masking, report industrial ablations or soften the attribution, and remove the scaling-law language. I would send it to review, but with the expectation of substantial revision.","headline":"Useful architecture work with real public speedups, but the industrial NDCG gains are not yet attributable to the design; needs revision before the headline claims are trusted.","tokens_in":20708,"tokens_out":2559,"would_cite":true,"duration_ms":28796,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative recommender can drop its query-key attention map and replace bucketed temporal bias with a learnable power function, matching or beating prior models while training faster.","keywords":["generative recommendation","sequential recommendation","efficient transformer","relative attention bias","query-key attention","attention-free token mixer","scaling law","large-scale recommendation"],"falsifier":"On the same industrial datasets, retrain FuXi-β with the query-key attention map added back (keeping FRAB), and retrain FuXi-α with per-model tuned hyperparameters and matched compute. If FuXi-β-with-QK performs as well as FuXi-β, or if a tuned FuXi-α matches FuXi-β, then the removal is not the cause. Additionally, plot train/validation NDCG curves for both models on industrial data: the overfitting story predicts FuXi-α's validation curve pulls away from training while FuXi-β's stays close.","tokens_in":19723,"feed_emoji":"⚡","tokens_out":7990,"duration_ms":75284,"temperature":0.7,"pith_summary":"This paper tries to establish that Transformer-like generative recommenders can be made both faster and more accurate by subtracting components, not adding them. It claims that the query-key attention map in self-attention is redundant for recommendation and often harmful, and that the relative attention bias—positional plus temporal—can carry the entire inter-item interaction. To replace the inefficient bucketed temporal bias used in earlier models, it introduces Functional Relative Attention Bias, a learnable power function $f(x)=a(1+x)^{-b}$, and to remove the query-key map it designs an Attention-Free Token Mixer. Applied to the previous FuXi-$\\alpha$ model, the new FuXi-$\\beta$ matches or slightly beats it on public benchmarks while training faster, and reports 27% to 47% higher NDCG@10 on two industrial-scale datasets, with the paper attributing that gain to reduced overfitting.","feed_headline":"Dropping query-key attention speeds up recommenders, boosts NDCG","feed_subtitle":"Replacing bucketed time bias with a learnable power function gives 27–47% better NDCG@10 on industrial data.","key_machinery":"The central mechanism is the Attention-Free Token Mixer (AFTM): an inter-item interaction layer that deletes the query and key matrices and instead multiplies SiLU-activated value projections by two attention maps—the positional Relative Attention Bias (RAB) and the new Functional Relative Attention Bias (FRAB). FRAB is the second load-bearing piece: a learnable power function $f(x)=a(1+x)^{-b}$ that replaces the bucketed temporal indexing of HSTU and FuXi-$\\alpha$, eliminating non-contiguous memory operations while providing a monotonically decaying recency weight.","core_discovery":"On the paper's own terms: the query-key attention map is not only unnecessary but counterproductive in autoregressive recommendation, because the relative attention bias terms already act as attention maps. Removing the query and key projections and mixing tokens through two bias maps—positional RAB and temporal FRAB—keeps or improves accuracy while cutting roughly half of the self-attention computation. FRAB replaces the bucketed, index-heavy temporal bias of HSTU and FuXi-$\\alpha$ with a scalar function $f(x)=a(1+x)^{-b}$ with learnable $a$ and $b$, avoiding non-contiguous memory access. The resulting model FuXi-$\\beta$ is at parity with FuXi-$\\alpha$ on MovieLens-1M and MovieLens-20M (e.g","pith_inferences":["Editorial inference: if query-key attention is systematically redundant in recommendation, the same removal could be applied to other sequential models (e.g., two-tower or LLM-based recommenders) to obtain similar speedups; the paper's own compatibility results make this a testable extension rather than a demonstrated result.","Editorial inference: the paper's overfitting explanation for the industrial gains is a post hoc hypothesis—no industrial ablations or training curves are shown. A direct test would be comparing train/validation gaps of FuXi-$\\alpha$ and FuXi-$\\beta$ on those datasets; a smaller gap for FuXi-$\\beta$ would support the claim, equal gaps would refute it.","Editorial inference: the function $f(x)=a(1+x)^{-b}$ is a power-law decay kernel, and the paper's own function sweep shows that only monotone-decaying functions (pow, exp) work well; this suggests the essential inductive bias is recency decay, linking this design to classical time-decay models of user interest.","Editorial inference: because public-dataset gains are near zero, the claimed superiority of removing the query-key map may be data-dependent; the public/industrial gap itself (data scale, sequence length, noise) is a natural target for follow-up study."],"forward_implications":["If the central claim holds, the default design of Transformer-like generative recommenders can be simplified: query-key attention can be removed entirely, and temporal bias can be a cheap scalar function, cutting self-attention cost roughly in half.","The speedups scale with model size: on MovieLens-20M the 8-layer FuXi-$\\beta$ trains 27% faster than FuXi-$\\alpha$, making larger (and better) models more affordable under fixed latency budgets.","The framework transfers: applying the same surgery to LLaMa-style and HSTU-style blocks reduces their training time on public datasets, though the paper reports HSTU-$\\beta$ loses accuracy, indicating the query-key map's role varies across architectures.","The scaling-law result on industrial data implies that removing attention components does not cap model growth; FuXi-$\\beta$ continues to improve in NDCG@10 as layers increase to 32.","On industrial-scale data, dropping the query-key map can yield large accuracy gains (27-47% NDCG@10), not just efficiency gains, if the paper's overfitting explanation is correct."],"supporting_citations":[{"why":"Supplies the bucketed relative attention bias and the generative recommender architecture that FuXi-α builds on; also the strongest baseline to beat.","marker":"[73]"},{"why":"The base model FuXi-α that FuXi-β modifies; source of the architecture being simplified.","marker":"[68]"},{"why":"Introduces the T5-style Relative Attention Bias (RAB) bucketing approach that FRAB replaces.","marker":"[42]"},{"why":"Shows that attention maps can be non-query-key forms (e.g., fixed or learnable constants), motivating the question of whether the query-key map is redundant.","marker":"[52]"},{"why":"Defines the self-attentive sequential recommendation paradigm and provides a key baseline model.","marker":"[26]"}],"fun_headline_variants":["Dropping query-key attention boosts recommenders 27–47% NDCG","No query-key attention: faster recommenders with higher NDCG","Remove key-query attention for 47% NDCG gain","Attention-free token mixer: FuXi-β is faster and better","Learning bias beats query-key attention for recommenders"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the 27-47% industrial NDCG gains are caused by removing the query-key attention map itself, not by the comparison setup—the paper applies identical hyperparameters to all baselines and offers no industrial ablation or training curves to demonstrate the overfitting mechanism; if FuXi-α is simply undertuned for industrial data, the headline superiority claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Dropping query-key attention boosts recommenders 27–47% NDCG","No query-key attention: faster recommenders with higher NDCG","Remove key-query attention for 47% NDCG gain","Attention-free token mixer: FuXi-β is faster and better","Learning bias beats query-key attention for recommenders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000826,"raw_usage":{"total_tokens":3519,"prompt_tokens":885,"completion_tokens":2634,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":2557}},"tokens_in":629,"tokens_out":2634,"duration_ms":18991,"temperature":1.0,"reasoning_tokens":2557,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:19:47.174255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same industrial datasets, retrain FuXi-β with the query-key attention map added back (keeping FRAB), and retrain FuXi-α with per-model tuned hyperparameters and matched compute. If FuXi-β-with-QK performs as well as FuXi-β, or if a tuned FuXi-α matches FuXi-β, then the removal is not the cause. Additionally, plot train/validation NDCG curves for both models on industrial data: the overfitting story predicts FuXi-α's validation curve pulls away from training while FuXi-β's stays close.","supporting_citations":[],"review_version":1}