REVIEW 3 major objections 5 minor 50 references
Personalizing Search Results Using Hierarchical RNN with Query-aware Attention
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper's central claim is that a hierarchical recurrent network that reads past sessions in order and weights them by the current query builds more accurate user profiles for search personalization than conventional click- or…
desk verdict The hierarchical RNN is a solid, honest search-personalization contribution; the query-aware attention is an exploratory add-on that the paper's own numbers show is not significant, so the title overclaims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a hierarchical recurrent neural network with query-aware attention. A low-level GRU consumes, for each query in a session, the concatenation of the query's word-vector representation and the average vector of documents the user clicked and dwelled on, which the paper calls satisfied clicks; its last hidden state is the session's short-term interest vector. Those vectors are fed in session order to a high-level GRU, whose hidden states after each session represent long-term interests. An MLP scores each past session state against the current query, producing softmax weights, and the weighted sum is the dynamic long-term profile. The profile is matched to candidate documents by cosine similarity through learned projection matrices, and the whole network is trained with LambdaRank using satisfied clicks as relevant documents.
What would settle it
Hold out a sample of test queries, replace the 30-second/last-click relevance labels with explicit human relevance judgments, and recompute MAP, MRR, and P@1 for HRNN+QA and SLTB+PTM; if the model's advantage shrinks or reverses, the reported gains are an artifact of the implicit labels rather than of better user profiling.
Extended reading notes
Core claim
The paper establishes that the sequential structure of search history, which queries appear in which order inside a session and which sessions precede the current one, carries personalization signal that aggregate click and topic features miss. It does so with a hierarchical recurrent network: a low-level GRU reads each session as a sequence of query and satisfied-click representations, and a high-level GRU reads the resulting session vectors in order. The last hidden state of the low level forms a short-term interest vector, while a query-aware attention-weighted sum of high-level states forms a dynamic long-term interest vector. Personalization scores are cosine similarities between these vectors and candidate documents, combined with position and click features and trained end-to-end with LambdaRank. The experiments on commercial search logs show the full model ranking above all baselines on MAP, MRR, P@1, average click position, and inverse document-pair improvement, with the authors reporting the gains over baselines are significant at p<0.01.
Load-bearing premise
The paper treats a click as a satisfied, relevant click when the user dwells on it more than 30 seconds or it is the last click in the session, and uses those clicks as ground truth for both training and all reported metrics; if dwell time and last-click status do not reliably indicate relevance, the measured gains could rest on noisy labels.
Editorial extensions
If this is right
- Personalization extends to non-repeated queries that have no click history, because the long-term topic profile can generalize where click-based features are empty.
- A single trained network replaces hand-designed temporal-decay and topic-aggregation rules, so re-ranking can be learned end-to-end from raw query and document text.
- Because attention weights highlight related past sessions, re-rankings become inspectable per user, showing which history drove the change.
- Gains concentrate on ambiguous, non-navigational queries, so a practical system can use query ambiguity to decide when personalized re-ranking is worth applying.
Reading between the lines
- A possible extension the paper only gestures at: replacing soft attention with discrete selection, such as reinforcement-learning gating, could keep long histories from spreading weight too thinly, since the paper itself notes attention over roughly 80 sessions per user is hard to train.
- The same hierarchical encoding should transfer to other logged user behaviors, such as recommender sessions, email triage, or assistant command histories, where the order and recency of interactions signal intent.
- If attention weights prove stable across repeated queries for a user, they could serve as user-facing explanations for re-ranked results, a use the paper does not evaluate.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hierarchical recurrent neural network (HRNN) with a query-aware attention mechanism for search result personalization. The low-level GRU encodes within-session query and SAT-click sequences into short-term interest vectors, and a high-level GRU aggregates session vectors into long-term interests. The query-aware attention computes weights over past session representations conditioned on the current query, producing a dynamic long-term profile. The model is trained with LambdaRank on click-through logs from a commercial search engine, and the experiments compare HRNN and HRNN+QA against P-Click, PTM, SLTB, and SLTB+PTM. The reported results show HRNN+QA achieving the best MAP, MRR, P@1, Avg. Click, and inverse document pair improvements, with additional analyses by click entropy, repeated/non-repeated queries, and query position within sessions, plus a single-user visualization of attention weights.
Significance. If fully supported, the paper would offer a useful deep-learning approach to search personalization that exploits sequential structure in query logs. The temporal heldout split by sessions is a sound evaluation choice, and the use of inverse document pairs as a supplementary metric addresses position bias. The HRNN model appears to improve over traditional personalization baselines by a large margin, and the ablation showing that HRNN also performs well on non-repeated queries is informative. However, the paper's headline contribution, the query-aware attention mechanism, is not supported by the reported statistics: the improvement of HRNN+QA over HRNN is not statistically significant, as the paper itself concedes in Section 5.1. The paper also reports only point estimates without standard deviations or confidence intervals, and the only direct evidence for attention is a single-user visualization. These gaps prevent the current manuscript from substantiating the title-level claim, though the HRNN contribution may be defensible.
major comments (3)
- [§5.1, Table 2] The central claim about query-aware attention is not supported by the reported experiments. In Table 2, HRNN+QA differs from HRNN by only 0.0028 in MAP (0.8017 vs. 0.7989), 0.0028 in MRR, 0.0028 in P@1, and 443 inverse document pairs, and Section 5.1 explicitly states that "such improvement is not statistically significant." Despite this, the title, abstract, Section 5.2 ("Table 2 shows that using this kind of dynamic user profiles improves personalization"), and the conclusion assert that the query-aware attention mechanism is effective and "able to highlight the important parts." Since query-aware attention is the paper's distinctive contribution, this is a load-bearing inconsistency. The authors should either provide statistically robust evidence for the attention component (e.g., multiple runs, paired tests, confidence intervals) or reframe the contribution around the hierarchical RNN and present attention as an exploratory extension.
- [§4.3, Table 2] All significance claims are based on point estimates from a single run, with no standard deviations, confidence intervals, or number of runs reported. Table 2 and Section 5.1 report paired t-test p-values for HRNN and HRNN+QA versus baselines, but a deep learning model's performance varies across random initializations and data subsampling; without variance information, the reader cannot assess the stability of the reported gains. The authors should report mean and standard deviation across multiple random seeds, and should apply the significance test to the HRNN+QA versus HRNN ablation specifically rather than only to the comparisons with baselines.
- [§4.1] The evaluation treats a click with dwelling time over 30 seconds, or the last click in a session, as a satisfied click and therefore as a relevant document for both LambdaRank training and all metrics. This is a standard approximation in the literature, but the paper does not validate these implicit labels against explicit relevance judgments. The concern is that SAT-clicks may be contaminated by position bias or noisy feedback, which would directly affect the measured MAP, MRR, P@1, and Avg. Click gains. The paper should at least discuss this limitation and, if feasible, provide a robustness check using the click-versus-skip inverse document pairs that it already computes in Section 4.1.
minor comments (5)
- [§5.1] In the observation list, "SLTB-PTM" appears to be a typo for the SLTB+PTM baseline; please correct for consistency.
- [Figure 2] The weight visualization lacks axis labels and a clear description of how the color intensity maps to numerical weights; please add a legend and define the displayed values.
- [§4.1] The Avg. Click metric is listed in Table 2 but is not defined in the evaluation section; please define it and explicitly state whether lower values are better.
- [§2 and §3.1.1] Section 2 states that the framework "only takes raw queries and documents with one hot representation of words as inputs," but Section 3.1.1 describes inputs as TF-IDF weighted average of word embeddings, with stopwords removed (Section 4.1). Please reconcile these descriptions.
- [Abstract] The abstract states "Significant improvement is observed" without specifying which comparisons are significant; please clarify that the significant gains are over the traditional baselines and that the attention gain over HRNN is not statistically significant.
Circularity Check
No material circularity: the reported personalization gains are measured on held-out sessions against external baselines, and no prediction is forced by construction.
full rationale
The paper's central derivation is self-contained. HRNN and HRNN+QA are trained end-to-end with LambdaRank on SAT-click labels derived from query logs and are evaluated on held-out future sessions, so the improvements over P-Click, PTM, SLTB, and SLTB+PTM in Table 2 are measured outcomes rather than consequences of the model definition. The query-aware attention module is learned during training, and the Figure 2 weight visualization is descriptive evidence, not a fitted reproduction of the headline metric. The paper explicitly concedes that the attention ablation's improvement over HRNN is not statistically significant; that is a limitation or correctness concern about the title-level claim, but it is not circularity, because the comparison remains an externally evaluated measurement. The self-citations to Dou et al. [11] for the P-Click baseline and to Bennett et al. [4] for the SAT-click convention are background definitions or comparison baselines, not load-bearing premises of the proposed model. No equation defines a predicted quantity in terms of a fitted parameter, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the chosen architecture. Therefore no load-bearing circular step is present, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (6)
- Word embedding dimension d_e =
300
- Short-term GRU hidden size d_s1 =
300
- Long-term GRU hidden size d_s2 =
600
- Attention MLP hidden size d_a =
1024
- Additional scoring MLP hidden size d_f =
64
- Learning rate lambda =
1e-3
assumptions (6)
- domain assumption SAT-click definition: clicks with dwelling time over 30 seconds, or the last click in a session, are treated as relevant for training and evaluation.
- domain assumption The logs were generated without personalization support, so clicks are not biased by prior personalization.
- domain assumption The last hidden state of a GRU over queries in a session represents the user's short-term interest.
- domain assumption A query-dependent weighted sum of high-level RNN hidden states represents the user's long-term interest.
- domain assumption TF-IDF weighted average of word vectors is an adequate representation of queries and documents.
- domain assumption The original rank position of a document can serve as the query-document relevance feature.
Cite this review
Pith. "Pith review of Personalizing Search Results Using Hierarchical RNN with Query-aware Attention." pith.science (2026). https://pith.science/paper/YWNSV6KA
@misc{pith2026190807600,
author = {Pith},
title = {Pith review of: Personalizing Search Results Using Hierarchical RNN with Query-aware Attention},
year = {2026},
howpublished = {\url{https://pith.science/paper/YWNSV6KA}},
note = {Machine review of arXiv:1908.07600}
}
read the original abstract
Search results personalization has become an effective way to improve the quality of search engines. Previous studies extracted information such as past clicks, user topical interests, query click entropy and so on to tailor the original ranking. However, few studies have taken into account the sequential information underlying previous queries and sessions. Intuitively, the order of issued queries is important in inferring the real user interests. And more recent sessions should provide more reliable personal signals than older sessions. In addition, the previous search history and user behaviors should influence the personalization of the current query depending on their relatedness. To implement these intuitions, in this paper we employ a hierarchical recurrent neural network to exploit such sequential information and automatically generate user profile from historical data. We propose a query-aware attention model to generate a dynamic user profile based on the input query. Significant improvement is observed in the experiment with data from a commercial search engine when compared with several traditional personalization models. Our analysis reveals that the attention model is able to attribute higher weights to more related past sessions after fine training.
Figures
Reference graph
Works this paper leans on
-
[1]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural ma- chine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
arXiv 2014
-
[2]
Paul N Bennett, Filip Radlinski, Ryen W White, and Emine Yilmaz. 2011. Inferring and using location metadata to personalize web search. In Proceedings of the SIGIR’2011. ACM, 135–144
work page 2011
-
[3]
Paul N Bennett, Krysta Svore, and Susan T Dumais. 2010. Classification-enhanced ranking. In Proceedings of the WWW’2010 . ACM, 111–120
work page 2010
-
[4]
Paul N Bennett, Ryen W White, Wei Chu, Susan T Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012. Modeling the impact of short-and long- term behavior on search personalization. In Proceedings of the SIGIR’2012 . ACM, 185–194
work page 2012
-
[5]
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. InProceedings of ICML’2005. ACM, 89–96
work page 2005
-
[6]
Fei Cai, Shangsong Liang, and Maarten De Rijke. 2014. Personalized document re-ranking based on bayesian probabilistic matrix factorization. In Proceedings of the SIGIR’2014. ACM, 835–838
work page 2014
-
[7]
Carman, Fabio Crestani, Morgan Harvey, and Mark Baillie
Mark J. Carman, Fabio Crestani, Morgan Harvey, and Mark Baillie. 2010. To- wards query log based personalization using topic models. In Proceedings of the CIKM’2010. 1849–1852
work page 2010
-
[8]
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP’2014. 1724–1734
work page 2014
Show all 50 references
-
[9]
Kevyn Collins-Thompson, Paul N Bennett, Ryen W White, Sebastian De La Chica, and David Sontag. 2011. Personalizing web search results by reading level. In Proceedings of the CIKM’2011 . ACM, 403–412
2011
-
[10]
Taylor, and Bill Ramsey
Nick Craswell, Onno Zoeter, Michael J. Taylor, and Bill Ramsey. 2008. An experi- mental comparison of click position-bias models. In WSDM’2008
2008
-
[11]
Zhicheng Dou, Ruihua Song, and Ji-Rong Wen. 2007. A large-scale evaluation and analysis of personalized search strategies. In WWW’2007. ACM, 581–590
2007
-
[12]
Salah El Hihi and Yoshua Bengio. 1996. Hierarchical recurrent neural networks for long-term dependencies. In NIPS’1996. 493–499
1996
-
[13]
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforce- ment Learning for Relation Classification From Noisy Data. In AAAI’2018
2018
-
[14]
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In CIKM’2016. ACM, 55–64
2016
-
[15]
Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krish- namurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring personalization of web search. In WWW’2013. ACM, 527–538
2013
-
[16]
Morgan Harvey, Fabio Crestani, and Mark J Carman. 2013. Building user profiles from topic models for personalised search. In CIKM’2013. ACM, 2309–2314
2013
-
[17]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation 9, 8 (1997), 1735–1780
1997
-
[18]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM’2013. ACM, 2333–2338
2013
-
[19]
Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In SIGKDD’2002. ACM, 133–142
2002
-
[20]
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay
-
[21]
Lin Li, Zhenglu Yang, Botao Wang, and Masaru Kitsuregawa. 2007. Dynamic adaptation strategies for long-term and short-term user profile to personalize search. Advances in Data and Web Management (2007), 228–240
2007
-
[22]
Xiujun Li, Chenlei Guo, Wei Chu, Ye-Yi Wang, and Jude Shavlik. 2014. Deep learning powered in-session contextual ranking using clickthrough data. In NIPS’2014
2014
-
[23]
Nicolaas Matthijs and Filip Radlinski. 2011. Personalizing web search using long term browsing history. In WSDM’2011. ACM, 25–34
2011
-
[24]
Bhaskar Mitra and Nick Craswell. 2017. Neural Models for Information Retrieval. arXiv preprint arXiv:1705.01509 (2017)
2017 arXiv
-
[25]
Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quin- ten McNamara, and others. 2018. Neural information retrieval: At the end of the early years. Information Retrieval Journal 21...
2018
-
[26]
Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. TASLP 24, 4 (2016), 694–707
2016
-
[27]
Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi
-
[28]
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models.. In AAAI’2016. 3776–3784
2016
-
[29]
Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In SIGIR’2015. ACM, 373–382
2015
-
[30]
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM’2014. ACM, 101–110
2014
-
[31]
Ahu Sieg, Bamshad Mobasher, and Robin Burke. 2007. Web search personalization with ontological user profiles. In CIKM’2007. ACM, 525–534
2007
-
[32]
Yang Song, Hongning Wang, and Xiaodong He. 2014. Adapting deep ranknet for personalized search. In WSDM’2014. ACM, 83–92
2014
-
[33]
David Sontag, Kevyn Collins-Thompson, Paul N Bennett, Ryen W White, Susan Dumais, and Bodo Billerbeck. 2012. Probabilistic models for personalizing web search. In WSDM’2012. ACM, 433–442
2012
-
[34]
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In CIKM’2015. ACM, 553–562
2015
-
[35]
Mirco Speretta and Susan Gauch. 2005. Personalized search based on user search histories. In Web Intelligence, 2005. Proceedings. The 2005 IEEE/WIC/ACM Interna- tional Conference on. IEEE, 622–628
2005
-
[36]
Jaime Teevan, Susan T Dumais, and Daniel J Liebling. 2008. To personalize or not to personalize: modeling queries with variation in user intent. In SIGIR’2018. ACM, 163–170
2008
-
[37]
Jaime Teevan, Daniel J Liebling, and Gayathri Ravichandran Geetha. 2011. Un- derstanding and predicting personal navigation. In WSDM’2011. ACM, 85–94
2011
-
[38]
Maksims Volkovs. 2015. Context models for web search personalization. arXiv preprint arXiv:1502.00527 (2015)
2015 arXiv
-
[39]
Thanh Vu, Dat Quoc Nguyen, Mark Johnson, Dawei Song, and Alistair Willis
-
[40]
Thanh Vu, Dawei Song, Alistair Willis, Son Ngoc Tran, and Jingfei Li. 2014. Improving search personalisation with dynamic group formation. In SIGIR’2014. 951–954
2014
-
[41]
Thanh Vu, Alistair Willis, Son N Tran, and Dawei Song. 2015. Temporal latent topic user profiles for search personalisation. In ECIR’2015. Springer, 605–616
2015
-
[42]
In ECIR’2017
Search personalization with embeddings. In ECIR’2017. Springer, 598–604
2017
-
[43]
White, W Ryen, Bennett, N Paul, Dumais, and T Susan. 2010. Predicting short- term interests using activity-based search context. (2010), 1009–1018
2010
-
[44]
Ryen W White, Peter Bailey, and Liwei Chen. 2009. Predicting user interests from contextual information. In SIGIR’2009. ACM, 363–370
2009
-
[45]
White, and Wei Chu
Hongning Wang, Xiaodong He, Ming Wei Chang, Yang Song, Ryen W. White, and Wei Chu. 2013. Personalized ranking model adaptation for web search. In SIGIR’2013. 323–332
2013
-
[46]
Qiang Wu, Chris JC Burges, Krysta M Svore, and Jianfeng Gao. 2008. Ranking, boosting, and model adaptation . Technical Report. Technical report, Microsoft Research
2008
-
[47]
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In SIGIR’2017
2017
-
[48]
Ryen W White, Wei Chu, Ahmed Hassan, Xiaodong He, Yang Song, and Hongn- ing Wang. 2013. Enhancing personalized search by mining and modeling task behavior. In WWW’2013. ACM, 1411–1420
2013
-
[2005]
In SI- GIR’2005
Accurately interpreting clickthrough data as implicit feedback. In SI- GIR’2005. 154–161
2005
-
[2017]
In RecSys’2017
Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks. In RecSys’2017. 130–137
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.