{"id":"e144e301-4de7-43ef-8fe7-41b46168145a","arxiv_id":"2507.03789","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding shifted query-context embeddings to the last attention layer's query position (plus the output head) improves next-item ranking in offline and online tests, though open-dataset validation uses a target-derived context proxy.","lead":"This paper studies how to feed the current query context, such as the browse category, into transformer-based sequential recommenders. It proposes two practical attention modifications and reports offline and online gains in ranking relevance and diversity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-dataset experiments leak the target label: using the next-item category as query context invalidates offline comparisons favoring Approach C and the RQ2 conclusion that historical context is unnecessary.","rationale":"The reader's weakest assumption is precisely the load-bearing issue. The open-dataset methodology deliberately substitutes the target item's category for query context; because that category is a direct attribute of the label being predicted, it leaks the answer into the input. This undermines the offline comparisons that are needed to differentiate Approach C from Approach B and from Approach A, and it also undermines the RQ2 claim that historical query context is not essential. The paper does provide genuine independent evidence for the broader value of query context: a production A/B test with statistically significant uplifts in engagement and revenue for Approach B, plus a plausible architectural analysis of training-serving mismatch in Section 5.3. The proposed Approach C is well-motivated and the reasoning about attention flow is internally coherent; I do not see a circularity or a missing proof inside the method itself. However, the specific central claim that Approach C is the best-performing or that it outperforms simply adding context to the final score rests on offline proprietary data without error bars and on open-dataset results that are contaminated by label leakage. The paper should either obtain a non-leaky query-context proxy for the open datasets, or clearly restrict its claims to the production online result, where only Approach B was tested significantly. This does not refute the paper's overall finding that query context helps; it means the evidence is weaker and more conditional than the abstract and conclusion suggest. Therefore I maintain the conditional verdict rather than accepting the paper's broader claims as established.","tokens_in":13702,"tokens_out":3702,"duration_ms":46131,"concrete_test":"Re-run the entire open-dataset pipeline of Table 3 and Table 4 on Taobao and Retailrocket with a non-leaky query-context proxy: use the category of the most recently interacted item (position i) as the query context for predicting item i+1, instead of the category of the target item (position i+1). Keep all model architectures, hyperparameters, masking rates, and evaluation metrics identical, and additionally report absolute NDCG@5 and NDCG@50 for a no-context baseline. If Approach C no longer outperforms Approaches A and B by comparable margins, or if the optimal masking rate no longer shows the reported concave trend, the open-dataset results are attributable to target-label leakage rather than to the proposed query-context integration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7 (Methodology) states: 'We use the next-item category as the query context.' The next-item category is a property of the exact item the model is trained and evaluated to predict, so it is label information. At evaluation, using this category as the input query context means the model is given the target's category before ranking, making the prediction task artificially easy. This affects every open-dataset result in Table 3 and Table 4. First, the reported gains of Approaches B and C over A may reflect how effectively each architecture exploits the leaked target category, not how well it integrates genuine query context. Second, the masking-rate trend (optimal masking 0.25) is confounded: partially masking the leaked signal can regularize an otherwise degenerate shortcut, so the observed concavity is not evidence about query-context masking in real deployments. Third, Table 4's conclusion that historical context is unnecessary is especially fragile: if the current context already equals the target's category, historical categories add little by construction. The proprietary offline results (Table 1) support the general value of query context in production, but they lack error bars and significance tests, and the only significant online comparison is for Approach B against the no-context baseline; the direct C-versus-B online comparison was not significant. Thus the central claim that Approach C improves ranking quality beyond adding context to the final score is not credibly supported by the open-dataset evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how to incorporate query context (e.g., browse category) into a transformer-based sequential recommender trained with causal language modeling. It identifies a temporal misalignment: the query context for the next item is not aligned with the item sequence at the position where the loss is computed. Three integration strategies are compared: adding the context as a separate feature outside the transformer (Approach A), adding shifted context embeddings to the first-layer input with random masking (Approach B), and adding shifted context embeddings only to the last layer's query position plus the output head (Approach C). The paper reports offline and online experiments on a large e-commerce platform and on two open datasets (Taobao, Retailrocket). The main positive evidence is a statistically significant online A/B test for Approach B against a no-context baseline. The paper argues that Approach C achieves similar gains while avoiding training-serving mismatch and misleading representations, and it concludes from open-dataset results that historical query context at inference is not essential.","tokens_in":13976,"tokens_out":2450,"duration_ms":31150,"significance":"If the central claims were fully supported, the paper would be a useful practical contribution: it identifies a real training-serving mismatch in fusing query context into transformer recommenders, proposes a computationally efficient integration point, and ships open-source code enabling reproduction. The reported online A/B test for Approach B is a genuine strength; it has confidence intervals and multiple engagement and revenue metrics. However, the open-dataset evaluation is undermined by a label-leakage problem: the query context is defined as the next-item category, which is the target label of the prediction task. This makes the offline comparisons in Tables 3 and 4 uninterpretable as evidence about query-context integration. The paper's central claim that Approach C improves ranking quality beyond baseline and beyond simpler context additions is not established by the provided evidence, because the online C-versus-B comparison was not statistically significant and the proprietary offline results lack error bars.","major_comments":[{"comment":"The open-dataset evaluation leaks the target label. The paper states 'We use the next-item category as the query context.' For a prediction at position i, the model is trained and evaluated to predict item x_{i+1}, and the query context c_{i+1} is exactly the category of that target item. Thus the model is given the target's category as an input before ranking, making the prediction task artificially easy. This invalidates the relative comparisons in Table 3 between Approaches A, B, and C, because any observed gains may simply reflect how effectively each architecture exploits the leaked category rather than how well it integrates genuine query context. The masking-rate trend (optimal masking 0.25) is also confounded: partial masking can regularize a degenerate shortcut, so the concavity is not evidence about query-context masking in deployment. The statement in Section 7 that a no-context model is not included because it would be 'inherently disadvantaged in this setting' confirms that the setting itself embeds the label into the input.","section":"Section 7, Methodology"},{"comment":"The conclusion that historical query context is not essential (RQ2) is especially fragile because of the same label leakage. If the current query context already equals the target item's category, historical categories add little information by construction; the null result in Table 4 is therefore not informative about whether historical query context matters when query context is a genuinely observed, non-label signal. This conclusion is load-bearing for the paper's stated RQ2 and for the practical guidance that historical context need not be stored or served.","section":"Section 7, Table 4"},{"comment":"The proprietary offline results in Table 1 are reported only as relative uplifts without confidence intervals, error bars, or significance tests. The differences between Approach C and Approach B are small in several rows (e.g., Browse Recall@500 +4.66% vs +4.69%, Search Recall@500 +2.32% vs +2.27%), so the reader cannot determine whether the reported ordering is noise. This matters because the paper uses Table 1 to argue that Approach C is the best-performing method.","section":"Section 6.1, Table 1"},{"comment":"The online A/B test provides statistically significant evidence only for Approach B against the no-context baseline. The paper explicitly states that testing Approach C against the new winner, Approach B, 'did not yield statistically significant improvements in the main metrics.' Therefore the core claim that Approach C improves ranking quality beyond a simpler approach that adds context to the final score is not supported by the online data. The paper's case for Approach C rests on its architectural properties and on the flawed open-dataset comparisons, not on a significant head-to-head result.","section":"Section 6.2, Online experiments"}],"minor_comments":[{"comment":"The sentence 'the projections in the previous layers,1 = 2,...,H -1' contains a typo; it should read 'h = 2, ..., H-1'.","section":"Section 5.3"},{"comment":"In the categorical cross-entropy loss, the term y^u_{t,x^u_{i+1}} uses an undefined index t; it should be y^u_{i,x^u_{i+1}} for consistency with the definition of y^u_{i,j}.","section":"Section 3.2, Loss equation"},{"comment":"The caption says the baseline is indicated by underline, but the table as rendered does not visually underline the row 'In the input without masking'. Please make the baseline formatting explicit or state the baseline in the caption.","section":"Section 7, Table 3"},{"comment":"The reference to 'contextual information' in the Taobao and Retailrocket datasets is imprecise: the datasets contain item categories rather than a recorded query or browse category chosen by the user. The approximation is stated, but the distinction between item category and observed query context should be emphasized earlier and more prominently.","section":"Section 7, first paragraph"},{"comment":"The negative sampling ratio is given as 0.5% of all classes with no further detail on how it interacts with the sampled softmax; a sentence on the sampling distribution or a reference to the standard log-uniform sampler would improve reproducibility.","section":"Section 6, Dataset description"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a better fit for a workshop or short industry paper than for a full archival journal at this stage. The label-leakage issue in the open-dataset evaluation is not a minor presentation problem; it directly undermines two of the paper's main conclusions (the relative ranking of approaches and RQ2). The production A/B test is a useful real-world data point, but it does not by itself establish the superiority of Approach C. A revision that honestly separates the valid production evidence from the invalid open-dataset interpretation, and that explicitly removes or re-scopes the RQ2 conclusion, would be a more defensible paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, there is a real industrial result here: a powered A/B test showing that adding query context to a transformer seq-rec model improves engagement, revenue, and diversity, with confidence intervals. That is worth having. Second, the open-dataset experiments are compromised: the paper uses the next-item category as the query context, and that category is the label the model is predicting. So the offline comparisons in Tables 3 and 4 do not measure what they claim.\n\nWhat's actually new: the paper systematically compares three ways to fuse query context into a causal transformer—outside the output head (A), injected into the input with random masking (B), and a variant (C) that adds shifted context only at the last layer's query position plus the output head. The design reasoning in Section 5.3 about avoiding training-serving mismatch and unnecessary conditioning of hidden states is clear and sensible. The code is released. Citation of Vlasov [22] for the design principle is self-referential, but the method is described in enough detail here that this is a minor point.\n\nThe soft spots are in the open evaluation. Saying 'we use the next-item category as the query context' is not a harmless approximation; it hands the model the answer. This makes every open-dataset number in Table 3 and Table 4 uninterpretable as evidence about query-context integration. The masking-rate trend in Table 3 could just reflect regularization of a leaky shortcut. The RQ2 conclusion that historical context is unnecessary is especially fragile: if the current context already equals the target's category, historical categories have little left to add. The proprietary offline table lacks error bars, and the online A/B test for Approach C versus B was not statistically significant. So the paper's strongest surviving claim is the Approach B online lift, not the superiority of Approach C.\n\nWho should read this: anyone working on sequential recommendation with side information, especially e-commerce, will get useful design vocabulary and a worked example of one production rollout. The temporal-misalignment framing is sound and general. But the open experiments need to be redone with a genuinely observed query context or clearly labeled as synthetic sanity checks, and the claims about Approach C and historical context need to be scaled back to what the A/B test supports.\n\nIt should go to peer review: the online experiment is significant enough to warrant referee time, and the research question is practical and relevant. As submitted, I would not accept without major revision of the offline evaluation.","headline":"A real online A/B lift for query context is the paper's solid core, but the open-dataset experiments leak the target label by using next-item category as the query context, so Approach C's offline edge and the RQ2 conclusion don't survive scrutiny.","tokens_in":14496,"tokens_out":2688,"would_cite":true,"duration_ms":29119,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Query context in the last attention layer improves recommendations","keywords":["sequential recommendation","query context","transformer","learning-to-rank","causal language modeling","training-serving mismatch","two-tower model","recommendation diversity"],"falsifier":"Train Approach C on an open dataset where the query context is an actually logged browsing category or search query that precedes the interaction and is not the target item's category, then compare with a no-context model and with the next-item-category proxy. If the real-context gains fall to zero or below the proxy-based gains, the central claim that fusing query context at the last-layer query position drives the improvement is not supported.","tokens_in":13492,"feed_emoji":"🛍️","tokens_out":5871,"duration_ms":60954,"temperature":0.7,"pith_summary":"This paper asks how a transformer-based sequential recommender can use query context - the category or search query under which a user is browsing - to predict the next item, when historical query context is unavailable at serving time. The authors compare three integration strategies and find that adding the shifted query context only at the query position of the last attention layer, plus adding it to the output scoring head, matches or exceeds the other placements while fully avoiding the training-serving mismatch. This placement, called Approach C, also avoids representing past items with information about later queries. The paper reports offline gains in recall and NDCG on a large-scale production ranking system and on open datasets, and an online A/B test of the input-side variant improved engagement, revenue, and recommendation diversity.","feed_headline":"Query context in the last attention layer improves recommendations","feed_subtitle":"Shifted query embeddings at the final query position beat input fusion and dodge the training-serving mismatch.","key_machinery":"The object that carries the argument is the shifted query-context embedding $\\boldsymbol{L}\\boldsymbol{C}$, where $\\boldsymbol{L}$ is an $N \\times (N+1)$ subdiagonal shift matrix bringing the context of the next item $c_{i+1}$ to position $i$. Approach C adds this term to the query projection of the last transformer layer, and adds the same shifted term to the output logits, while leaving all keys and values clean. That placement is what allows the model to be trained with historical context and served with only the current context, since no future information ever enters the hidden-state stream.","core_discovery":"On its own terms, the paper's central claim is that query context should enter a causal transformer at exactly one controlled point: the query projection of the last layer, together with the output head. Writing the shifted context matrix as $\\boldsymbol{L}\\boldsymbol{C}$ with $\\boldsymbol{L}$ a subdiagonal shift, the method uses $\\boldsymbol{Q}^{(H)} = (\\boldsymbol{X}^{(H)} + \\boldsymbol{L}\\boldsymbol{C})\\boldsymbol{W}^{(H)}_Q$ while leaving $\\boldsymbol{K}^{(H)}$ and $\\boldsymbol{V}^{(H)}$ unmodified; the same shifted context is added to the logits. The paper argues this fuses past items with the current query context inside attention without propagating next-item information through earlier layers or through keys and values, and it reports offline uplifts on the production Browse and Search ranking use cases (up to +6.56% NDCG@500 for Browse) and online gains in engagement, orders, and diversity, with the A/B test of the input-side variant showing +1.6% engagement and +0.7% revenue. On Taobao and Retailrocket, Approach C is within a small margin of the best input-side variant while remaining immune to the training-serving mismatch.","pith_inferences":["In our reading, the next-item-category proxy is the vulnerable spot: it makes the query context informative by construction, so the open-dataset deltas should be re-checked with a logged query that precedes the interaction.","The last-layer query-position idea is a general recipe for any causal transformer that must consume an action-level context at scoring time, not only recommenders; it may transfer to other sequence labeling and ranking tasks with a final-step context.","Because Approach C needs no historical context at serving, it could sidestep privacy and feature-store constraints in production without architectural surgery; this is a consequence the paper states qualitatively but does not quantify.","A natural extension is to feed item-category embeddings at the input layer (as the paper itself suggests) to recover the historical-context sequence signal while still using Approach C for the current context."],"forward_implications":["Using query context as a separate feature (Approach A) already improves Recall@500 and NDCG@500 in the production Browse and Search models, but fusing it inside attention (Approaches B and C) yields larger gains.","Approach C reaches production offline uplifts of +4.66% Recall@500 and +4.65% NDCG@500 for Browse and +2.32% and +3.04% for Search, matching the best input-side variant while avoiding the training-serving mismatch.","On Taobao and Retailrocket, optimized Approach B (masking rate 0.25) and Approach C are close in quality, and both outperform no-mask input fusion when only current context is available at inference.","Historical query context at inference did not improve results over current-context-only serving in either the open datasets or the production setup, so the paper concludes it may not be essential.","The online A/B test of the input-side variant improved engagement by +1.6% with 95% CI [1.4, 1.9]%, revenue by +0.7% [0.3, 1.2]%, and cut impressions on non-diverse pages by 66%."],"supporting_citations":[{"why":"Supplies the SASRec-style causal language modeling architecture and training procedure used as the base model.","marker":"[10]"},{"why":"Provides the alternative masked-language-model transformer baseline for sequential recommendation.","marker":"[19]"},{"why":"Represents prior category-aware input-fusion methods whose concerns motivate the new placement.","marker":"[4]"},{"why":"Motivates the privacy and feature-store constraints that make historical query context unavailable at inference.","marker":"[13]"},{"why":"Contributes the idea of injecting context at the last layer's query position to avoid contaminating the hidden-state stream.","marker":"[22]"},{"why":"Defines the two-tower user/item architecture that the production ranking model is built on.","marker":"[7]"},{"why":"Supplies the mixed negative sampling technique used in the production sampled-softmax training.","marker":"[26]"},{"why":"Provides the TransAct random time-window masking technique that motivates the extensibility concern with input fusion.","marker":"[23]"}],"fun_headline_variants":["Query context at last attention layer boosts ranking","Shifted query context in final layer avoids mismatch","Single attention point for query context improves recommendations","Last-layer query injection avoids input fusion mismatch","Query context only at last layer's query projection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The open-dataset experiments use the category of the next item as the query context, but that category is the very label the model is trained to predict, so using it as input leaks the answer; if this proxy is not a faithful substitute for a genuinely observed query context, the open-dataset comparisons do not support the paper's conclusions.","fun_headline_variants_meta":{"raw":{"variants":["Query context at last attention layer boosts ranking","Shifted query context in final layer avoids mismatch","Single attention point for query context improves recommendations","Last-layer query injection avoids input fusion mismatch","Query context only at last layer's query projection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001224,"raw_usage":{"total_tokens":5044,"prompt_tokens":971,"completion_tokens":4073,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":4005}},"tokens_in":587,"tokens_out":4073,"duration_ms":30636,"temperature":1.0,"reasoning_tokens":4005,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:02:20.736674+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Approach C on an open dataset where the query context is an actually logged browsing category or search query that precedes the interaction and is not the target item's category, then compare with a no-context model and with the next-item-category proxy. If the real-context gains fall to zero or below the proxy-based gains, the central claim that fusing query context at the last-layer query position drives the improvement is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SASRec-style causal language modeling architecture and training procedure used as the base model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the privacy and feature-store constraints that make historical query context unavailable at inference."},{"cited_title":"Efficient slot labelling","cited_arxiv_id":"2401.09343","evidence_quote":"Contributes the idea of injecting context at the last layer's query position to avoid contaminating the hidden-state stream."}],"review_version":1}