REVIEW 3 major objections 5 minor 33 references
Learning to Ask: Question-based Sequential Bayesian Product Search
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A short sequence of yes/no entity questions can outperform six product-search baselines.
desk verdict A useful heuristic for interactive product search, but the Bayesian derivation is wrong, so the optimality claims should be removed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the candidate version space $U_l$ together with a Dirichlet belief $P_n$ over the user's preference distribution $\pi^*$, where a Dirichlet is a family of distributions over probability vectors that stays in the same family after multinomial observations. After a yes/no question about entity $e$, the version space is cut to products whose entity indicator matches the answer, and the belief is updated as $P_{n+1}(\pi)\propto\pi(d)P_n(\pi)$, implemented as a single Dirichlet with one pseudo-count added to each surviving product (Eq. 3). The expected preference $\pi^*_n(d)=E_{\pi\sim P_n}[\pi(d)]$ produces the ranked list. Question selection (Eq. 6) chooses the entity minimizing $|\sum_{d\in U_l}(2\,\mathbf{1}\{e(d)=1\}-1)\pi^*(d)| - \gamma R(e)$, which is Generalized Binary Search's probability-mass split combined with a reward $R(e)$ learned offline as the average rank improvement of the target product when that entity is asked. The offline duet training supplies both the prior belief $P_t(\pi)$ and the rewards $R_t(e)$ from historical products in the same topic.
What would settle it
Run QSBPS on a small artificial topic, say five products and ten entities, with a known target product, and compare the next-question choice and top-ranked product under the paper's single-Dirichlet pseudo-count update against an exact mixture-of-Dirichlet posterior; any divergence in question choice or top rank shows the approximation changes the algorithm's behaviour.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a system can locate the target product by asking a short sequence of yes/no questions about the presence of extracted entities in the target's documents, and that learning which entities to ask from other users' purchase histories makes this practical with limited data. QSBPS maintains a Dirichlet belief over the distribution from which the target product is drawn, and after each answer adds a pseudo-count to every product consistent with the answer. The next question is the entity that best balances the remaining probability mass on the two sides of the yes/no split, with a learned reward term for how much that entity has helped in training. The paper reports that QSBPS achieves the highest NDCG and Recall@5 among Random, LSE, TranSearch, ALSTP, PMMN, and SBS at every tested question count from 5 to 30, and that with only about five questions it already exceeds the static baselines. The claimed upshot is that entity questions close the query-document vocabulary gap more effectively than improved query representations alone.
Load-bearing premise
The load-bearing premise is that after each yes/no answer the belief can be collapsed into one Dirichlet distribution, a distribution over possible product-probability vectors, by adding one pseudo-count to every product consistent with the answer; the exact posterior is a mixture of Dirichlets, and the ranking and question-choice rules rely on the collapsed form.
Editorial extensions
If this is right
- With fewer than five questions, QSBPS already surpasses the static query-product matching baselines LSE, TranSearch, and ALSTP on the two Amazon categories tested, so entity questions can substitute for much of the ranking model's work.
- The optimal weight on the learned question reward decreases as more questions are allowed; reward matters most when only a few questions can be asked, and can be dropped once many questions are available.
- Adding user reviews to product metadata improves every metric at every question count beyond the first, because reviews contain entities that discriminate products better than descriptions do.
- Without any training, QSBPS still converges to near-perfect performance when 25-30 questions are allowed, but the duet training is what makes it effective with few questions (e.g., 5-10).
- Under simulated noisy answers, the noise-tolerant variant with an entity-specific error model outperforms the static baselines after about four questions even with wrong answers, so the method does not require perfectly reliable users.
Reading between the lines
- The single-Dirichlet update in Eq. (3) is an approximation: the exact posterior after a yes/no answer is a mixture of Dirichlets, one per product still in the version space. A natural test the paper does not report is to replace the pseudo-count update with an exact mixture update on a small collection and check whether question choice or ranking changes.
- The framework is only instantiated with surface entities, but the same duet-training machinery could in principle be applied to attribute-value pairs, relations, or image features; whether the gains persist with those question pools is an open empirical question.
- The user simulation assumes users know whether an entity appears in the target product, possibly with a fixed error rate; real users may answer 'not sure' or refuse to answer. The paper leaves a user study to future work, and the practical benefit depends on how such answers are incorporated.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes QSBPS, an interactive product search method that asks users a sequence of yes/no questions about whether entities appear in the documents of the target product. The method consists of an offline 'duet training' phase that learns a system belief over product relevance and per-entity question rewards from historical purchase data, and an online phase that selects questions via an objective combining GBS-style probability-mass splitting with the learned reward, updating a Dirichlet belief after each answer. Experiments on two Amazon product domains with simulated users compare QSBPS against Random, LSE, TranSearch, ALSTP, PMMN, and SBS, reporting higher NDCG and Recall@5 when 5-30 questions are asked. The paper also studies the influence of user reviews, the effect of duet training, and robustness to noisy answers.
Significance. If the empirical results hold, the paper would show that a handful of entity-presence questions can outperform state-of-the-art static and conversational product search baselines, which is a practically interesting result. The paper ships source code, uses a standard public dataset, and includes a useful noise-tolerance analysis. However, the claimed Bayesian derivation is not correct for the stated observation model, and the printed training algorithm is internally inconsistent, so the empirical claim currently rests on an unsupported and possibly misdescribed method. The significance is therefore contingent on a corrected and verifiable description of the algorithm.
major comments (3)
- [Section 3.2, Eqs. (2)-(4)] The update in Eq. (2) is not Bayes' rule for the stated observation model. A user's answer to 'does the target product contain entity e?' has likelihood P(answer=yes|π) = Σ_{d:e(d)=1} π(d) and P(answer=no|π) = Σ_{d:e(d)=0} π(d). Eq. (2), P_{n+1}(π) ∝ π(d)P_n(π), is the likelihood of directly observing the target product d, not of observing an entity answer. Consequently, Eq. (3) does not follow from Bayes' rule: conditioning a Dirichlet prior on a yes/no entity answer yields a mixture of Dirichlet components, one per product consistent with the answer, and this mixture is not a single Dirichlet distribution. Eq. (4) is therefore not the posterior mean under the actual observation process. Since Eq. (6) is described as a GBS-style optimal question-selection rule and the method is named 'Bayesian Product Search,' this is a load-bearing error. The authors should either derive the exact mixture posterior and use it, or explicitly state that the pseudo-count update in Eq. (3) is a heuristic approximation and remove the exact-Bayesian and GBS-optimality claims.
- [Algorithm 1, lines 7-8] The offline training loop updates the belief as P_{n+1}(π) ∝ π(d)P_n(π) for every entity e, with no dependence on e or on any question answer. As written, the trained system belief Pt(π) is simply a Dirichlet with pseudo-counts added for each training product in D_t repeated |E| times; it is never conditioned on answers to entity questions. Moreover, line 8 computes the reward R_d(e) from the rank improvement caused by asking entity e, but if the belief update does not depend on e, the rank of target product d after the update is identical for every e, so R_d(e) would be constant across entities and the 'question reward training' would learn nothing. This contradicts the text in Section 3.2, which says the posterior is obtained 'after every question for the target training product d is being answered.' The pseudocode must be corrected to update the belief using the answer to entity e (e.g., via the indicator vector Z_j in Eq. (3)) and to compute I_after from the ranking after that entity-specific update. If the released implementation follows the printed algorithm, the experimental results cannot be attributed to learned question rewards.
- [Section 4.1.3 and Section 5] The evaluation relies entirely on simulated users who answer with perfect knowledge of entity presence (or with a fixed/tf-based noise model), and the paper acknowledges in the conclusions that a user study is future work. This is a properly stated limitation, but the abstract and Section 4.6 phrase the results as a general improvement ('greatly improve the performance of product search'), which should be qualified as applying to the simulated setup. The authors should also report variance or significance tests for the Table 6 comparisons, since the claim of 'highest effectiveness' is based on point estimates without error bars, and the margins over SBS at 15+ questions are sometimes small.
minor comments (5)
- [Section 3.3, Eq. (8)] The noise-rate function h(e) = 1/(2(1+TFavg(e))) is explicitly admitted to be ad-hoc; this is acceptable as a first model, but the paper should at least cite or compare to alternative noise models, since the noise-tolerance results in Table 5 depend on this choice.
- [Section 3.2, notation] The subscript for the question index is inconsistent: Eq. (1) uses n, while Section 3.2 and Algorithm 2 use l; the text also says 'in the n-th question' and 'the n-th question' in the same paragraph. Please unify the notation.
- [Eq. (5) and Algorithm 1] The reward definition uses |U| without specifying whether U is the version space before or after the question, and the tie-breaking rule ('worst ranking index') is described only in prose; define these quantities precisely in the equation or surrounding text.
- [Table 6] For SBS, the NDCG at 5 questions is 0.001 (Home & Kitchen) and 0 (Clothing), which is surprisingly low; please clarify whether SBS is given any training signal in this setup and whether this baseline is being compared fairly.
- [General] There are several typos, e.g., 'the it h entry' after Eq. (4), 'choosen' before Eq. (6), and 'the n-th question' in Section 3.2; a careful proofread is needed.
Circularity Check
No circularity: QSBPS is evaluated on a held-out split against an aligned but separately fitted reward objective; the derivational concerns in the Bayesian update are correctness issues, not circular reductions.
full rationale
The paper's central empirical claim (Section 4.6, Table 6) is that QSBPS outperforms six baselines on held-out Amazon product-search data. The training phase (Algorithm 1) fits a system belief and per-entity rewards from training products, and the online phase (Algorithm 2) selects entities via Eq. (6), which combines a GBS-style probability split with the learned reward R(e). The reward in Eq. (5) is defined as the rank improvement of the target product on the training split, so it is aligned with the MRR/Recall@k/NDCG metrics reported in Table 6. That is a supervised training objective, not a prediction of the held-out metric: the optimal gamma and beta are chosen on the validation split, and the reported numbers are test-split results (Section 4.1.1), so the comparison is not forced by construction. The noise model h(e) in Eq. (8) is explicitly labeled 'ad-hoc' and a user study is deferred to future work; this is an acknowledged modeling limitation, not a circular step. The only serious concern from a close reading is mathematical rather than circular: for a yes/no entity question the likelihood is a sum of pi(d) over products consistent with the answer, whereas Eqs. (2)-(3) and the pseudocode in Algorithms 1 and 2 update as if the target product itself were observed. This would undermine the 'Bayesian' optimality interpretation, but it does not make the reported effectiveness reduce by construction to a fitted quantity or to a self-citation. The self-citations ([30], [22]) are background and not load-bearing. Therefore, no circularity is found.
Assumptions & free parameters
free parameters (4)
- gamma (γ) =
Varies from 0 to 1 depending on domain and number of questions; e.g., H&K: 1, 0.5, 0.1, 0.1, 0.1, 0; CTJ: 1, 0.2, 0.8…
- beta (β) =
1 on Home & Kitchen, 0.8 on Clothing, Shoes & Jewelry
- alpha0 (α0) =
1 for all products
- Noise rate function h(e) =
h(e) = 1 / (2 * (1 + TF_avg(e)))
assumptions (5)
- domain assumption The target product is drawn i.i.d. from a fixed but unknown multinomial distribution over products.
- domain assumption Users answer yes/no questions about entity presence with perfect knowledge (or with a specified error rate h(e)).
- domain assumption The topic (subcategory) of a user's query is known in advance.
- ad hoc to paper The posterior after a yes/no answer is a single Dirichlet with pseudo-counts added to all products in the version space.
- domain assumption TAGME entity annotations accurately reflect the characteristics of products relevant to users.
Cite this review
Pith. "Pith review of Learning to Ask: Question-based Sequential Bayesian Product Search." pith.science (2026). https://pith.science/paper/K7U2J73A
@misc{pith2026190811733,
author = {Pith},
title = {Pith review of: Learning to Ask: Question-based Sequential Bayesian Product Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/K7U2J73A}},
note = {Machine review of arXiv:1908.11733}
}
read the original abstract
Product search is generally recognized as the first and foremost stage of online shopping and thus significant for users and retailers of e-commerce. Most of the traditional retrieval methods use some similarity functions to match the user's query and the document that describes a product, either directly or in a latent vector space. However, user queries are often too general to capture the minute details of the specific product that a user is looking for. In this paper, we propose a novel interactive method to effectively locate the best matching product. The method is based on the assumption that there is a set of candidate questions for each product to be asked. In this work, we instantiate this candidate set by making the hypothesis that products can be discriminated by the entities that appear in the documents associated with them. We propose a Question-based Sequential Bayesian Product Search method, QSBPS, which directly queries users on the expected presence of entities in the relevant product documents. The method learns the product relevance as well as the reward of the potential questions to be asked to the user by being trained on the search history and purchase behavior of a specific user together with that of other users. The experimental results show that the proposed method can greatly improve the performance of product search compared to the state-of-the-art baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W. Bruce Croft. 2017. Learning a Hierarchical Embedding Model for Personalized Product Search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’17) . ACM, New York, NY, USA, 645–654. https://doi.org/10.1145/3077136.3080813
arXiv 2017
-
[2]
Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Célia Nunes, and Adam Jatowt. 2018. A Text Feature Based Automatic Keyword Extraction Method for Single Documents. In Advances in Information Retrieval. Springer International Publishing, Cham, 684–691
work page 2018
-
[3]
Yihong Chen, Bei Chen, Xuguang Duan, Jian-Guang Lou, Yue Wang, Wenwu Zhu, and Yong Cao. 2018. Learning-to-Ask: Knowledge Acquisition via 20 Questions. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18) . ACM, New York, NY, USA, 1216–1225
work page 2018
-
[4]
Gordon V. Cormack and Maura R. Grossman. 2014. Evaluation of Machine- learning Protocols for Technology-assisted Review in Electronic Discovery. In Proceedings of the 37th international ACM SIGIR conference on Research & develop- ment in information retrieval (SIGIR ’14) . ACM, New York, NY, USA, 153–162
work page 2014
-
[5]
Gordon V Cormack and Maura R Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. arXiv preprint arXiv:1504.06868 (2015)
arXiv 2015
-
[6]
Huizhong Duan, ChengXiang Zhai, Jinxing Cheng, and Abhishek Gattani. 2013. A probabilistic mixture model for mining and analyzing product search log. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management (CIKM ’13) . ACM, New York, NY, USA, 2179–2188
work page 2013
-
[7]
Huizhong Duan, ChengXiang Zhai, Jinxing Cheng, and Abhishek Gattani. 2013. Supporting Keyword Search in Product Database: A Probabilistic Approach.Proc. VLDB Endow. 6, 14 (Sept. 2013), 1786–1797
work page 2013
-
[8]
Paolo Ferragina and Ugo Scaiella. 2010. TAGME: On-the-fly Annotation of Short Text Fragments (by Wikipedia Entities). In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM ’10) . ACM, New York, NY, USA, 1625–1628. https://doi.org/10.1145/1871437.1871689
arXiv 2010
Show all 33 references
-
[9]
Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yinglong Wang, Jun Ma, and Mohan Kankanhalli. 2019. Attentive Long Short-Term Preference Modeling for Person- alized Product Search. ACM Transactions on Information Systems (TOIS) 37, 2 (2019), 19
2019
-
[10]
Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Xin-Shun Xu, and Mohan Kankan- halli. 2018. Multi-modal Preference Modeling for Product Search. In Proceedings of the 26th ACM International Conference on Multimedia (MM ’18) . ACM, New York, NY, USA, 1865–1873. https://doi.org/10.1145...
2018
-
[11]
TE Gupta. 2017. Keyword extraction: a review.International Journal of Engineering Applied Sciences and Technology 2, 4 (2017), 215–220
2017
-
[12]
Katja Hofmann, Shimon Whiteson, and Maarten de Rijke. 2013. Balancing ex- ploration and exploitation in listwise and pairwise online learning to rank for information retrieval. Information Retrieval 16, 1 (2013), 63–90
2013
-
[13]
Huang Hu, Xianchao Wu, Bingfeng Luo, Chongyang Tao, Can Xu, Wei Wu, and Zhan Chen. 2018. Playing 20 Question Game with Policy-Based Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing . Association for Computation...
2018 doi
-
[14]
Soon Chong Johnson Lim, Ying Liu, and Wing Bun Lee. 2010. Multi-facet product information search and retrieval using semantically annotated product family ontology. Information Processing & Management 46, 4 (2010), 479 – 493
2010
-
[15]
Yuanhua Lv and ChengXiang Zhai. 2009. Adaptive Relevance Feedback in Infor- mation Retrieval. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM ’09) . ACM, New York, NY, USA, 255–264
2009
-
[16]
Julian McAuley, Rahul Pandey, and Jure Leskovec. 2015. Inferring Networks of Substitutable and Complementary Products. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’15). ACM, New York, NY, USA, 785–794. https://doi....
2015
-
[17]
Robert Nowak. 2008. Generalized binary search. In 2008 46th Annual Allerton Conference on Communication, Control, and Computing . IEEE, 568–574
2008
-
[18]
Hadas Raviv, Oren Kurland, and David Carmel. 2016. Document Retrieval Using Entity-Based Language Models. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’16). ACM, New York, NY, USA, 65–74. https://doi....
2016
-
[19]
Khapra, and Sachindra Joshi
Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Joshi. 2017. Generating Natural Language Question-Answer Pairs from a Knowledge Graph Using a RNN Based Question Generation Model. In Proceedings of the 15th Con- ference of the European Chapter of the Association fo...
2017
-
[20]
Jennifer Rowley. 2000. Product search in e-shopping: a review and research propositions. Journal of consumer marketing 17, 1 (2000), 20–35
2000
-
[21]
Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’18) . ACM, New York, NY, USA, 235–244
2018
-
[22]
Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2016. Learning Latent Vector Spaces for Product Search. In Proceedings of the 25th ACM Inter- national on Conference on Information and Knowledge Management (CIKM ’16) . ACM, New York, NY, USA, 165–174. https://do...
2016
-
[23]
Damir Vandic, Flavius Frasincar, and Uzay Kaymak. 2013. Facet Selection Algo- rithms for Web Product Search. In Proceedings of the 22Nd ACM International Conference on Information & Knowledge Management (CIKM ’13) . ACM, New York, NY, USA, 2327–2332. https://doi.org/10.1145/25...
2013
-
[24]
Damir Vandic, Jan-Willem van Dam, and Flavius Frasincar. 2012. Faceted product search powered by the Semantic Web. Decision Support Systems 53, 3 (2012), 425 – 437
2012
-
[25]
Zheng Wen, Branislav Kveton, Brian Eriksson, and Sandilya Bhamidipati. 2013. Sequential Bayesian Search. In Proceedings of the 30th International Conference on Machine Learning (Proceedings of Machine Learning Research) , Vol. 28. PMLR, Atlanta, Georgia, USA, 226–234
2013
-
[26]
Chenyan Xiong and Jamie Callan. 2015. EsdRank: Connecting Query and Docu- ments Through External Semi-Structured Data. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (CIKM ’15). ACM, New York, NY, USA, 951–960. https://doi.or...
2015
-
[27]
Chenyan Xiong, Jamie Callan, and Tie-Yan Liu. 2017. Word-Entity Duet Repre- sentations for Document Ranking. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’17). ACM, New York, NY, USA, 763–772. https:/...
2017
-
[28]
Hamed Zamani and W Bruce Croft. 2018. Joint modeling and optimization of search and recommendation. arXiv preprint arXiv:1807.05631 (2018)
2018 arXiv
-
[29]
Bruce Croft
Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018. Towards Conversational Search and Recommendation: System Ask, User Respond. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM ’18) . ACM, New York, NY, ...
2018
-
[30]
Jie Zou, Dan Li, and Evangelos Kanoulas. 2018. Technology Assisted Reviews: Finding the Last Few Relevant Documents by Asking Yes/No Questions to Re- viewers. In The 41st International ACM SIGIR Conference on Research & Develop- ment in Information Retrieval (SIGIR ’18) . ACM,...
2018
-
[31]
J. Zou, L. Xu, M. Yang, M. Yan, D. Yang, and X. Zhang. 2016. Duplication Detec- tion for Software Bug Reports based on Topic Model. In 2016 9th International Conference on Service Science (ICSS) . 60–65. https://doi.org/10.1109/ICSS.2016.16
2016 doi
-
[32]
Jie Zou, Ling Xu, Mengning Yang, Xiaohong Zhang, and Dan Yang. 2017. Towards comprehending the non-functional requirements through Developers’ eyes: An exploration of Stack Overflow using topic analysis. Information and Software Technology 84 (2017), 19 – 32. https://doi.org/1...
2017 doi
-
[33]
Jie ZOU, Ling XU, Mengning YANG, Xiaohong ZHANG, Jun ZENG, and Sachio HIROKAWA. 2016. Automated Duplicate Bug Report Detection Using Multi- Factor Analysis. IEICE Transactions on Information and Systems E99.D, 7 (2016), 1762–1775. https://doi.org/10.1587/transinf.2016EDP7052
2016 doi
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.