{"id":"f05e10d2-6d72-449f-8407-6535028fb2a1","arxiv_id":"1908.07600","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A hierarchical recurrent network with query-aware attention learns short- and long-term user profiles from click logs and reports improved personalization on a commercial search engine log.","lead":"Search engines often return the same results for the same query even when users mean different things. This paper proposes a neural model that learns a user's short- and long-term interests from past query sessions and re-ranks results accordingly, reporting improved click-based metrics on a commercial search engine log.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The query-aware attention component has no statistically significant support: HRNN+QA differs from HRNN by only 0.0028 MAP, and the paper itself reports the difference is not significant, undermining the title's central claim.","rationale":"The paper's central claim is best read as two separable claims: (1) a hierarchical RNN can build better user profiles than traditional click- or topic-based methods, and (2) the query-aware attention mechanism is what makes this work dynamic and effective. The first claim is supported by significant p<0.01 comparisons against SLTB and SLTB+PTM. The second claim is the paper's main novelty and appears in the title, abstract, and contribution list, but the reported ablation between HRNN and HRNN+QA is not statistically significant. This is not an external-consensus disagreement or a suspicion about data quality; it is an internal inconsistency within the paper's own results. The reader's weakest assumption focused on SAT-click label noise, which is a legitimate external-validity concern but one that applies roughly equally to all compared methods and follows standard IR practice. The attention-significance problem is more directly load-bearing for the headline claim: even if the SAT-click labels are accepted as ground truth, the paper's distinctive contribution has no demonstrated effect. The only quantitative evidence for attention is 443 additional inverse pairs fixed, which the authors themselves say is not significant, plus a single-user visualization that cannot support a general claim. A per-user bootstrap or clustered paired test would settle whether the small observed delta is real; if it is not, the paper should be revised to frame HRNN without attention as the primary contribution. I keep the verdict UNCHANGED because the paper was already CONDITIONAL, and this concern further supports the conditionality rather than moving to a different verdict.","tokens_in":16325,"tokens_out":6672,"duration_ms":70493,"concrete_test":"Run a per-user cluster-robust paired bootstrap on the HRNN versus HRNN+QA differences in MAP, MRR, P@1, and #Better over all test sessions, and report 95% confidence intervals. If the intervals contain zero, the attention claim should be removed from the title and abstract or supported by a properly powered experiment; if the intervals exclude zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 explicitly concedes that the query-aware attention module improves HRNN by only 443 inverse document pairs and states that \"such improvement is not statistically significant.\" Table 2 shows HRNN+QA versus HRNN: MAP 0.8017 versus 0.7989, MRR 0.8135 versus 0.8107, and P@1 0.7067 versus 0.7039. The distinctive contribution claimed in the title, abstract, and Section 1 is the query-aware attention mechanism, yet this ablation is within the noise of the experiment. The strong significance claims at p<0.01 in Table 2 and Section 5.1 refer to comparisons with the baseline systems, not to the attention ablation. The only direct evidence offered for attention is the single-user weight visualization in Figure 2, which is anecdotal and not a statistical test. This is an internal inconsistency: the abstract and conclusion assert that the attention model highlights important past sessions, while the paper's own quantitative results do not establish any reliable benefit from attention. HRNN alone may still be a valid contribution, but the title-level claim about query-aware attention is not supported by the reported measurements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hierarchical recurrent neural network (HRNN) with a query-aware attention mechanism for search result personalization. The low-level GRU encodes within-session query and SAT-click sequences into short-term interest vectors, and a high-level GRU aggregates session vectors into long-term interests. The query-aware attention computes weights over past session representations conditioned on the current query, producing a dynamic long-term profile. The model is trained with LambdaRank on click-through logs from a commercial search engine, and the experiments compare HRNN and HRNN+QA against P-Click, PTM, SLTB, and SLTB+PTM. The reported results show HRNN+QA achieving the best MAP, MRR, P@1, Avg. Click, and inverse document pair improvements, with additional analyses by click entropy, repeated/non-repeated queries, and query position within sessions, plus a single-user visualization of attention weights.","tokens_in":16568,"tokens_out":4644,"duration_ms":493276,"significance":"If fully supported, the paper would offer a useful deep-learning approach to search personalization that exploits sequential structure in query logs. The temporal heldout split by sessions is a sound evaluation choice, and the use of inverse document pairs as a supplementary metric addresses position bias. The HRNN model appears to improve over traditional personalization baselines by a large margin, and the ablation showing that HRNN also performs well on non-repeated queries is informative. However, the paper's headline contribution, the query-aware attention mechanism, is not supported by the reported statistics: the improvement of HRNN+QA over HRNN is not statistically significant, as the paper itself concedes in Section 5.1. The paper also reports only point estimates without standard deviations or confidence intervals, and the only direct evidence for attention is a single-user visualization. These gaps prevent the current manuscript from substantiating the title-level claim, though the HRNN contribution may be defensible.","major_comments":[{"comment":"The central claim about query-aware attention is not supported by the reported experiments. In Table 2, HRNN+QA differs from HRNN by only 0.0028 in MAP (0.8017 vs. 0.7989), 0.0028 in MRR, 0.0028 in P@1, and 443 inverse document pairs, and Section 5.1 explicitly states that \"such improvement is not statistically significant.\" Despite this, the title, abstract, Section 5.2 (\"Table 2 shows that using this kind of dynamic user profiles improves personalization\"), and the conclusion assert that the query-aware attention mechanism is effective and \"able to highlight the important parts.\" Since query-aware attention is the paper's distinctive contribution, this is a load-bearing inconsistency. The authors should either provide statistically robust evidence for the attention component (e.g., multiple runs, paired tests, confidence intervals) or reframe the contribution around the hierarchical RNN and present attention as an exploratory extension.","section":"§5.1, Table 2"},{"comment":"All significance claims are based on point estimates from a single run, with no standard deviations, confidence intervals, or number of runs reported. Table 2 and Section 5.1 report paired t-test p-values for HRNN and HRNN+QA versus baselines, but a deep learning model's performance varies across random initializations and data subsampling; without variance information, the reader cannot assess the stability of the reported gains. The authors should report mean and standard deviation across multiple random seeds, and should apply the significance test to the HRNN+QA versus HRNN ablation specifically rather than only to the comparisons with baselines.","section":"§4.3, Table 2"},{"comment":"The evaluation treats a click with dwelling time over 30 seconds, or the last click in a session, as a satisfied click and therefore as a relevant document for both LambdaRank training and all metrics. This is a standard approximation in the literature, but the paper does not validate these implicit labels against explicit relevance judgments. The concern is that SAT-clicks may be contaminated by position bias or noisy feedback, which would directly affect the measured MAP, MRR, P@1, and Avg. Click gains. The paper should at least discuss this limitation and, if feasible, provide a robustness check using the click-versus-skip inverse document pairs that it already computes in Section 4.1.","section":"§4.1"}],"minor_comments":[{"comment":"In the observation list, \"SLTB-PTM\" appears to be a typo for the SLTB+PTM baseline; please correct for consistency.","section":"§5.1"},{"comment":"The weight visualization lacks axis labels and a clear description of how the color intensity maps to numerical weights; please add a legend and define the displayed values.","section":"Figure 2"},{"comment":"The Avg. Click metric is listed in Table 2 but is not defined in the evaluation section; please define it and explicitly state whether lower values are better.","section":"§4.1"},{"comment":"Section 2 states that the framework \"only takes raw queries and documents with one hot representation of words as inputs,\" but Section 3.1.1 describes inputs as TF-IDF weighted average of word embeddings, with stopwords removed (Section 4.1). Please reconcile these descriptions.","section":"§2 and §3.1.1"},{"comment":"The abstract states \"Significant improvement is observed\" without specifying which comparisons are significant; please clarify that the significant gains are over the traditional baselines and that the attention gain over HRNN is not statistically significant.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's main gap is the unsupported claim about query-aware attention. The authors are transparent about the non-significance in Section 5.1, so this is fixable by either strengthening the evidence or reframing the contribution. I would also ask the editor to require a statement about code/data availability, as the absence of released code and the commercial nature of the dataset limit reproducibility. No concerns about the fitness of the topic for this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: the hierarchical RNN is the real contribution, and the query-aware attention is not. The stress-test note is accurate. Section 5.1 says the attention gain over HRNN is only 443 inverse document pairs and \"not statistically significant.\" The title and abstract advertise query-aware attention as the central novelty, but the paper's own measurements do not support that framing. Figure 2 is a single-user weight visualization, which is anecdotal. Read as \"HRNN for search personalization, with a speculative attention mechanism,\" the paper is solid; read as \"query-aware attention works,\" it is not.\n\nWhat is actually new: a two-level GRU over query and session sequences to build short- and long-term user profiles for re-ranking, trained with LambdaRank. That combination is not in the cited prior work. The evaluation is better than many personalization papers: held-out sessions split in time order, five baselines including the strong SLTB+PTM, inverse-document-pair metrics that address position bias, and informative breakdowns by click entropy and repeated/non-repeated queries. The ablation of short-term versus long-term RNN at different session positions is a nice diagnostic and gives some real insight into when each profile helps.\n\nSoft spots, in proportion. The attention effect is within noise; the body admits it, but the abstract and conclusion do not. That mismatch needs fixing. Table 2 gives point estimates and p-values without standard deviations or confidence intervals; with large test sets, p<0.01 against baselines is not the same as a meaningful effect size. The SAT-click definition (dwell time over 30 seconds or last click) is used as ground truth for training and all metrics, without validation against explicit judgments; this is standard in the area, but it means the reported gains are about click prediction, not relevance. Using original position as a feature is a reasonable proxy but worth a controlled comparison. No code or data is available, which is expected for commercial logs but makes the attention result even harder to assess. Citation pattern looks clean; related work is appropriately cited, and there is no circularity in the evaluation.\n\nWho this is for: people working on session-based personalization and neural learning-to-rank. The HRNN result and the position analysis deserve attention; the attention component should be treated as exploratory. I would accept this for peer review. The main flaw is in framing, not in the underlying experiment. A serious referee should ask for the claims to be scaled back, error bars or confidence intervals added, and an explicit statement that the attention component is not statistically supported.","headline":"The hierarchical RNN is a solid, honest search-personalization contribution; the query-aware attention is an exploratory add-on that the paper's own numbers show is not significant, so the title overclaims.","tokens_in":17091,"tokens_out":2527,"would_cite":true,"duration_ms":120957,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a hierarchical recurrent network that reads past sessions in order and weights them by the current query builds more accurate user profiles for search personalization than conventional click- or…","keywords":["search results personalization","hierarchical recurrent neural network","query-aware attention","user profile modeling","click-through data","learning to rank","LambdaRank","session modeling"],"falsifier":"Hold out a sample of test queries, replace the 30-second/last-click relevance labels with explicit human relevance judgments, and recompute MAP, MRR, and P@1 for HRNN+QA and SLTB+PTM; if the model's advantage shrinks or reverses, the reported gains are an artifact of the implicit labels rather than of better user profiling.","tokens_in":16131,"feed_emoji":"🔍","tokens_out":10303,"duration_ms":92029,"temperature":0.7,"pith_summary":"The central claim is that the order of a user's past queries and sessions carries personalization signal that aggregate click and topic features miss. The paper proposes a two-level recurrent network in which a low level reads each session as a sequence of queries and satisfied clicks, and a high level reads the resulting session summaries in order; a query-aware attention layer then weights past sessions by their relevance to the current query, producing a dynamic long-term profile. On two months of logs from a commercial search engine, the full model (HRNN+QA) reaches MAP 0.8017 against 0.7830 for the strongest conventional baseline (SLTB+PTM), with the authors reporting significance at p<0.01. A sympathetic reader would take the paper as evidence that sequence-aware, query-conditioned profiles are a better basis for re-ranking than static user models.","feed_headline":"Search history in order beats click- and topic-based profiles","feed_subtitle":"The order of past queries helps a search engine re-rank results for each user, beating static click- and topic-based profiles.","key_machinery":"The central machinery is a hierarchical recurrent neural network with query-aware attention. A low-level GRU consumes, for each query in a session, the concatenation of the query's word-vector representation and the average vector of documents the user clicked and dwelled on, which the paper calls satisfied clicks; its last hidden state is the session's short-term interest vector. Those vectors are fed in session order to a high-level GRU, whose hidden states after each session represent long-term interests. An MLP scores each past session state against the current query, producing softmax weights, and the weighted sum is the dynamic long-term profile. The profile is matched to candidate documents by cosine similarity through learned projection matrices, and the whole network is trained with LambdaRank using satisfied clicks as relevant documents.","core_discovery":"The paper establishes that the sequential structure of search history, which queries appear in which order inside a session and which sessions precede the current one, carries personalization signal that aggregate click and topic features miss. It does so with a hierarchical recurrent network: a low-level GRU reads each session as a sequence of query and satisfied-click representations, and a high-level GRU reads the resulting session vectors in order. The last hidden state of the low level forms a short-term interest vector, while a query-aware attention-weighted sum of high-level states forms a dynamic long-term interest vector. Personalization scores are cosine similarities between these vectors and candidate documents, combined with position and click features and trained end-to-end with LambdaRank. The experiments on commercial search logs show the full model ranking above all baselines on MAP, MRR, P@1, average click position, and inverse document-pair improvement, with the authors reporting the gains over baselines are significant at p<0.01.","pith_inferences":["A possible extension the paper only gestures at: replacing soft attention with discrete selection, such as reinforcement-learning gating, could keep long histories from spreading weight too thinly, since the paper itself notes attention over roughly 80 sessions per user is hard to train.","The same hierarchical encoding should transfer to other logged user behaviors, such as recommender sessions, email triage, or assistant command histories, where the order and recency of interactions signal intent.","If attention weights prove stable across repeated queries for a user, they could serve as user-facing explanations for re-ranked results, a use the paper does not evaluate."],"forward_implications":["Personalization extends to non-repeated queries that have no click history, because the long-term topic profile can generalize where click-based features are empty.","A single trained network replaces hand-designed temporal-decay and topic-aggregation rules, so re-ranking can be learned end-to-end from raw query and document text.","Because attention weights highlight related past sessions, re-rankings become inspectable per user, showing which history drove the change.","Gains concentrate on ambiguous, non-navigational queries, so a practical system can use query ambiguity to decide when personalized re-ranking is worth applying."],"supporting_citations":[{"why":"Defines the SLTB baseline and the satisfied-click rule (dwell time over 30 seconds or last click in session) that the paper adopts for training and evaluation.","marker":"[4]"},{"why":"Provides LambdaRank, the learning-to-rank algorithm used to train the whole hierarchical framework.","marker":"[5]"},{"why":"Supplies the GRU cell used as the basic recurrent unit in both the low-level and high-level RNNs.","marker":"[8]"},{"why":"Introduces the hierarchical recurrent neural network concept that motivates the two-level architecture for modeling long-term dependencies.","marker":"[12]"},{"why":"Provides the PTM latent-topic user profile model used as a baseline for comparison.","marker":"[16]"},{"why":"Defines the inverse document-pair evaluation used to compute #Better and P-Improve, the reliable-improvement metrics in the experiments.","marker":"[20]"},{"why":"Source of temporal latent-topic user profiles and the satisfied-click labeling convention the paper follows.","marker":"[41]"},{"why":"Introduces the attention mechanism that the query-aware weighting extends to past search sessions.","marker":"[1]"}],"fun_headline_variants":["Query order in search history boosts personalization","Order matters: RNN ranks search results better","Sequential search history improves ranking personalization","Hierarchical RNN with attention personalizes search","Query order beats click counts for search personalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats a click as a satisfied, relevant click when the user dwells on it more than 30 seconds or it is the last click in the session, and uses those clicks as ground truth for both training and all reported metrics; if dwell time and last-click status do not reliably indicate relevance, the measured gains could rest on noisy labels.","fun_headline_variants_meta":{"raw":{"variants":["Query order in search history boosts personalization","Order matters: RNN ranks search results better","Sequential search history improves ranking personalization","Hierarchical RNN with attention personalizes search","Query order beats click counts for search personalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3261,"prompt_tokens":921,"completion_tokens":2340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2271}},"tokens_in":537,"tokens_out":2340,"duration_ms":15049,"temperature":1.0,"reasoning_tokens":2271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:02:14.567398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a sample of test queries, replace the 30-second/last-click relevance labels with explicit human relevance judgments, and recompute MAP, MRR, and P@1 for HRNN+QA and SLTB+PTM; if the model's advantage shrinks or reverses, the reported gains are an artifact of the implicit labels rather than of better user profiling.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SLTB baseline and the satisfied-click rule (dwell time over 30 seconds or last click in session) that the paper adopts for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LambdaRank, the learning-to-rank algorithm used to train the whole hierarchical framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GRU cell used as the basic recurrent unit in both the low-level and high-level RNNs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the hierarchical recurrent neural network concept that motivates the two-level architecture for modeling long-term dependencies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PTM latent-topic user profile model used as a baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of temporal latent-topic user profiles and the satisfied-click labeling convention the paper follows."}],"review_version":1}