REVIEW 3 major objections 7 minor 40 references
QEQR: An Exploration of Query Expansion Methods for Question Retrieval in CQA Services
T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that expanding a user's question with words from the most semantically similar questions, selected by a contextualized language model and with the question's central words withheld from expansion, raises retrieval…
desk verdict A careful, modest empirical study whose headline 1.8% gain is plausible but currently impossible to verify because the centrality-based expansion is under-specified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is an expanded question language model built by interpolating three components: the original question's maximum-likelihood language model, a language model estimated from the k most ELMo-similar questions (k=5), and a pseudo-relevance feedback language model over the first two retrieved questions (weights 0.3, 0.3, and 0.2 respectively). ELMo, a pre-trained bidirectional language model, supplies context-dependent word vectors that are length-scale averaged to represent each question; cosine similarity between these vectors selects the expansion questions. A fixed-point term-centrality algorithm marks one or two words as the question's central intent-bearing words, and those words are removed from the expansion process: their vectors are excluded from question representations, and for two-word cases the expansion sets are intersected. This selectivity is what turns the word- and question-similarity expansions into consistently significant gains.
What would settle it
Re-running the experiment with the centrality regularizer constant c swept over a grid, or evaluating on a held-out split larger than the 630 test questions used, would show whether the 0.7428 MAP holds or regresses toward the LM-PRF level of 0.7328.
Extended reading notes
Core claim
The central claim is that expanding an input question with terms drawn from the most semantically similar questions, selected using ELMo-based contextualized representations and with the question's central words excluded from expansion, improves question retrieval performance on the 1,260-question test set to a MAP of 0.7428, a statistically significant relative improvement of about 1.8% over the TR-LM baseline (0.7291). The same experiments support a second claim: feedback questions chosen by ELMo similarity are more useful for expansion than feedback questions chosen by pseudo-relevance feedback alone, because ELMo captures semantic and syntactic similarity rather than just lexical overlap.
Load-bearing premise
The reported advantage depends on the algorithm that picks a question's central words behaving on short question titles the same way it behaves on the long web queries it was designed for; the paper omits the algorithm's full details and never reports a key internal constant, so if that transfer fails the gain could disappear.
Editorial extensions
If this is right
- Question retrieval systems that already use KL-divergence language models could adopt this expansion recipe without retraining, since the gains come from retrieval-time interpolation rather than learned rankers.
- The finding that ELMo-selected questions outperform PRF-selected questions as feedback suggests that building feedback sets from semantic similarity rather than first-pass lexical retrieval is a sufficient condition for the improvement.
- Withholding central words from expansion is shown to help across all three expansion families (word-level, whole-question, and question-level), indicating that selective expansion is transferable beyond the specific ELMo configuration.
- The reported 1.8% relative MAP gain is small but consistent, and it is achieved on top of already strong language-model and translation-model baselines, meaning the method addresses residual lexical gap rather than a weak baseline.
Reading between the lines
- A natural next test the paper leaves implicit is whether the same selective-expansion idea carries over to transformer-based dense retrievers; the gains here are demonstrated on lexical matching models, so the margin may shrink or grow under a different retrieval backbone.
- The paper's own Table 2 suggests a testable refinement: when two words jointly carry a question's intent, expanding either one alone may distort the intent less than expanding both, and dropping both may be safest; this could be evaluated by a controlled ablation.
- Because the centrality algorithm's IDF regularizer constant c is never reported, a reproducibility check should fix c on a development set before trusting the selective-expansion component; this omission makes the selectivity contribution partially under-specified as published.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses the lexical gap problem in community question answering (CQA) question retrieval by exploring query expansion methods. It adapts two existing word-embedding-based expansion methods (expAL and expKuzi), proposes a question-similarity-based expansion method using ELMo contextualized representations (expELMo), and augments it with pseudo-relevance feedback (expELMoPRF). A further contribution is selective expansion, which identifies 'central' terms using the Paik-Oard centrality method and excludes them from the expansion process, yielding expAL-centrality, expKuzi-centrality, expELMo-centrality, and exp-ELMoPRF-centrality. On the Zhang et al. test collection, exp-ELMoPRF-centrality reports MAP 0.7428, a claimed statistically significant 1.8% relative improvement over the best non-expansion baseline TR-LM (MAP 0.7291). The paper also claims that ELMo-selected similar questions provide better feedback than PRF-selected questions.
Significance. If the best method is fully specified and reproducible, this is a useful empirical exploration: it systematically compares several expansion strategies on a public relevance-judged collection and provides evidence that excluding central words during expansion can improve retrieval. A strength is the use of an external test collection and a held-out development split for parameter tuning. The headline improvement is modest (1.8% relative over TR-LM), so the paper's value depends on the reliability of the significance claim and on the completeness of the method description. The under-specification of the centrality component currently prevents independent reproduction, which is the main obstacle to accepting the empirical claims.
major comments (3)
- [Section 3.3, Eq. (16)] The centrality algorithm is explicitly omitted ('we omit the full details of the algorithm'), and the value of the IDF regularization constant c in Eq. (16) is never reported nor shown to be tuned in Section 4.2. Because the final term importance I(qi) = A(qi) * didf(qi) determines which terms are designated central and hence excluded from expansion in the best-performing method, the headline result (exp-ELMoPRF-centrality, Table 5) cannot be independently reproduced or sensitivity-checked. Please provide the full algorithm (or a precise pointer to the exact equations in Paik and Oard [37]) and report the value of c; if c was fixed without tuning, please provide a sensitivity analysis over a range of c values.
- [Section 3.3, Eq. (19)] For questions with two central words, the final ELMo feedback set is specified as the intersection of two k-nearest-neighbor sets obtained after excluding each central word in turn. The paper does not state how an empty intersection, or an intersection smaller than k, is handled. Since Table 2 explicitly shows two-central-word cases, the expansion procedure is undefined for a potentially non-negligible subset of the 630 test queries. Please specify the fallback rule (e.g., union, use only one of the two sets, or reduce k) and, if possible, report how many test queries have two central words and how many of those yield empty or sub-k intersections.
- [Section 4.3, Tables 4 and 5] The claim that all improvements in Table 5 are 'statistically significant ... using t-test with 95% confidence level' is not accompanied by the test variant (paired or unpaired), the test statistic, or the p-value. With only 630 test queries and a 1.8% relative gain, the significance claim is load-bearing for the paper's main conclusion. Please report the details of the test, including the unit of analysis (e.g., per-query average precision), degrees of freedom, and the exact p-value (or a confidence interval) for exp-ELMoPRF-centrality versus TR-LM, and ideally for the other pairwise comparisons claimed to be significant.
minor comments (7)
- [Section 3, Eq. (1)] The KL-divergence scoring formula appears garbled: 'log pseen (w|θ QC ) α dp (w|C)' seems to be missing the fraction bar and the denominator's α_d term. Please rewrite the equation cleanly so that the smoothing notation is unambiguous.
- [Section 2.3] The text groups '[24] and [27]' as 'word2vec models'; reference [27] is Pennington et al.'s GloVe, which is a different count-based embedding model. Please correct the wording to 'word embedding models' or otherwise distinguish the two.
- [Section 3.3] There is a typo 'euq. 7' in the sentence before Eq. (18); it should be 'Eq. (7)'. Also, the name 'paik et al.' should be capitalized as 'Paik et al.' for consistency.
- [Table 2] The table caption does not explain the blue and red color coding used to distinguish the central word before and after applying IDF. Since the paper may be read in grayscale, please add textual markers (e.g., asterisks or superscripts) to the table.
- [Section 4.2] The text says α values in expAL were tested in 'period [0.2, 0.4]'; 'period' should be 'range'. Also, please clarify whether the α and β values in expELMoPRF-centrality were tuned jointly or independently, and over what ranges.
- [Section 4.3] The claim that expELMo outperforms LM-PRF ('we can conclude that using questions ... are more accurate compared to the questions that are suggested by LM-PRF') is based on MAP 0.7375 versus 0.7328 without a reported significance test. Either add a significance test for this comparison or soften the conclusion.
- [Section 4.1] The paper does not state whether all baselines and proposed methods use the same preprocessing (e.g., stopword removal, stemming, Dirichlet prior μ). Please report these settings so that the comparisons are fully interpretable.
Circularity Check
No circularity: the evaluation uses an external relevance-judged test set, and the expansion methods do not consume relevance labels or fitted test metrics.
full rationale
No circular reasoning was found. The retrieval evaluation uses the external relevance-judged test collection of Zhang et al. [16], and none of the proposed expansion methods use relevance labels. expELMo and expELMoPRF build the expanded language model from unsupervised ELMo-based similar questions and pseudo-relevance feedback questions (Eqs. 12-15), interpolated with the original question language model and scored by KL-divergence (Eq. 1). The selective-expansion component (Sec. 3.3) identifies central words by the external Paik-Oard centrality algorithm [37] over pseudo-relevance feedback documents and then excludes those words from expansion; this changes the expansion set but does not define the retrieval target. Hyperparameters (e.g., k=5, alpha=0.3, beta=0.2, lambda=0.65) are tuned only on the development half of the test set, so the reported test MAP is not fitted by construction. The unreported IDF regularization constant c and the undefined handling of an empty intersection of nearest-neighbor sets are reproducibility and robustness concerns, but they are not instances of circular derivation.
Assumptions & free parameters
free parameters (10)
- alpha (expAL) =
0.4
- k (expAL) =
2
- v (expKuzi) =
9
- lambda (expKuzi) =
0.65
- alpha (expELMo) =
0.3
- alpha, beta (expELMoPRF) =
0.3, 0.2
- k (expELMo and expELMoPRF) =
5
- c (IDF regularization) =
not reported
- PRF feedback count =
2
- Centrality EM iterations and PRF doc count =
12 iterations; 10 PRF docs
assumptions (4)
- standard math KL-divergence retrieval with Dirichlet-prior smoothing is the relevance scoring model.
- domain assumption ELMo embeddings transfer to short CQA questions and capture semantic similarity.
- domain assumption Paik-Oard centrality correctly identifies words whose expansion would change the question intent.
- domain assumption The Zhang et al. relevance judgments and Yahoo! L6 data are representative for evaluating CQA question retrieval.
Cite this review
Pith. "Pith review of QEQR: An Exploration of Query Expansion Methods for Question Retrieval in CQA Services." pith.science (2026). https://pith.science/paper/FGWYRA5M
@misc{pith2026241115530,
author = {Pith},
title = {Pith review of: QEQR: An Exploration of Query Expansion Methods for Question Retrieval in CQA Services},
year = {2026},
howpublished = {\url{https://pith.science/paper/FGWYRA5M}},
note = {Machine review of arXiv:2411.15530}
}
read the original abstract
CQA services are valuable sources of knowledge that can be used to find answers to users' information needs. In these services, question retrieval aims to help users with their information needs by finding similar questions to theirs. However, finding similar questions is obstructed by the lexical gap that exists between relevant questions. In this work, we target this problem by using query expansion methods. We use word-similarity-based methods, propose a question-similarity-based method and selective expansion of these methods to expand a question that's been submitted and mitigate the lexical gap problem. Our best method achieves a significant relative improvement of 1.8\% compared to the best-performing baseline without query expansion.
Reference graph
Works this paper leans on
-
[37]
A fixed-point method for weighting terms in verbose informational queries
Jiaul H Paik and Douglas W Oard. “A fixed-point method for weighting terms in verbose informational queries”. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM. 2014, pp. 131–140
work page 2014
-
[1]
A comprehensive survey a nd classification of approaches for community question answering
Ivan Srba and Maria Bielikova. “A comprehensive survey a nd classification of approaches for community question answering”. In: ACM Transactions on the Web (TWEB) 10.3 (2016), p. 18
work page 2016
-
[2]
Finding simi lar questions in large question and answer archives
Jiwoon Jeon, W Bruce Croft, and Joon Ho Lee. “Finding simi lar questions in large question and answer archives”. In: Proceedings of the 14th ACM in- ternational conference on Information and knowledge manag ement. ACM. 2005, pp. 84–90
work page 2005
-
[3]
Retrieval models for question and answer archives
Xiaobing Xue, Jiwoon Jeon, and W Bruce Croft. “Retrieval models for question and answer archives”. In: Proceedings of the 31st annual interna- tional ACM SIGIR conference on Research and development in i nforma- tion retrieval. ACM. 2008, pp. 475–482
work page 2008
-
[4]
Amit Singh. “Entity based q&a retrieval”. In: Proceedings of the 2012 Joint conference on empirical methods in natural language p rocessing and computational natural language learning . Association for Computational Linguistics. 2012, pp. 1266–1277
work page 2012
-
[5]
Using key concepts in a tr anslation model for retrieval
Jae Hyun Park and W Bruce Croft. “Using key concepts in a tr anslation model for retrieval”. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retr ieval. ACM. 2015, pp. 927–930
work page 2015
-
[6]
Learning the latent topics for question ret rieval in commu- nity qa
Li Cai et al. “Learning the latent topics for question ret rieval in commu- nity qa”. In: Proceedings of 5th international joint conference on Natur al Language Processing. 2011, pp. 273–281
work page 2011
-
[7]
Question-answer topic model for que stion retrieval in community question answering
Zongcheng Ji et al. “Question-answer topic model for que stion retrieval in community question answering”. In: Proceedings of the 21st ACM in- ternational conference on Information and knowledge manag ement. ACM. 2012, pp. 2471–2474
work page 2012
Show all 40 references
-
[8]
A topic clustering approach to findi ng similar ques- tions from large question and answer archives
Wei-Nan Zhang et al. “A topic clustering approach to findi ng similar ques- tions from large question and answer archives”. In: PloS one 9.3 (2014), e71511
2014
-
[9]
The use of categorization information in l anguage models for question retrieval
Xin Cao et al. “The use of categorization information in l anguage models for question retrieval”. In: Proceedings of the 18th ACM conference on Information and knowledge management . 2009, pp. 265–274
2009
-
[10]
Approaches to exploring category inform ation for question retrieval in community question-answer archives
Xin Cao et al. “Approaches to exploring category inform ation for question retrieval in community question-answer archives”. In: ACM Transactions on Information Systems (TOIS) 30.2 (2012), p. 7
2012
-
[11]
Towards faster and better retriev al models for question search
Guangyou Zhou et al. “Towards faster and better retriev al models for question search”. In: Proceedings of the 22nd ACM international conference on Information & Knowledge Management . ACM. 2013, pp. 2139–2148
2013
-
[12]
Group non-negative matrix factor ization with natu- ral categories for question retrieval in community question answer archives
Guangyou Zhou et al. “Group non-negative matrix factor ization with natu- ral categories for question retrieval in community question answer archives”. In: Proceedings of COLING 2014, the 25th International Confere nce on Computational Linguistics: Technical Papers . 2014, pp. 89–98
2014
-
[13]
Searching questions by identifyi ng question topic and question focus
Huizhong Duan et al. “Searching questions by identifyi ng question topic and question focus”. In: Proceedings of ACL-08: HLT. 2008, pp. 156–164. 14 Ghafourian et al
2008
-
[14]
Segmentation of multi-sentence questi ons: towards effec- tive question retrieval in cqa services
Kai Wang et al. “Segmentation of multi-sentence questi ons: towards effec- tive question retrieval in cqa services”. In: Proceedings of the 33rd inter- national ACM SIGIR conference on Research and development i n infor- mation retrieval. ACM. 2010, pp. 387–394
2010
-
[15]
Modeling and learning conti nuous word embedding with metadata for question retrieval
Guangyou Zhou and J Huang. “Modeling and learning conti nuous word embedding with metadata for question retrieval”. In: IEEE Transactions on Knowledge and Data Engineering 1 (2017), pp. 1–1
2017
-
[16]
Learning distributed representation s of data in com- munity question answering for question retrieval
Kai Zhang et al. “Learning distributed representation s of data in com- munity question answering for question retrieval”. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mi ning. ACM. 2016, pp. 533–542
2016
-
[17]
Concept embedded convolutional se mantic model for question retrieval
Pengwei Wang et al. “Concept embedded convolutional se mantic model for question retrieval”. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining . ACM. 2017, pp. 395–403
2017
-
[18]
Neural matching models for question ret rieval and next question prediction in conversation
Liu Yang et al. “Neural matching models for question ret rieval and next question prediction in conversation”. In: arXiv preprint arXiv:1707.05409 (2017)
2017 arXiv
-
[19]
The vocabulary problem in human -system com- munication
George W. Furnas et al. “The vocabulary problem in human -system com- munication”. In: Communications of the ACM 30.11 (1987), pp. 964–971
1987
-
[20]
On relevance, p robabilistic indexing and information retrieval
Melvin Earl Maron and John Larry Kuhns. “On relevance, p robabilistic indexing and information retrieval”. In: Journal of the ACM (JACM) 7.3 (1960), pp. 216–244
1960
-
[21]
Relevance feedback in information re trieval
Joseph Rocchio. “Relevance feedback in information re trieval”. In: The Smart retrieval system-experiments in automatic document processing (1971), pp. 313–323
1971
-
[22]
Relevance Feedback and Other Query Modification Tech- niques
Donna Harman. Relevance Feedback and Other Query Modification Tech- niques. 1992
1992
-
[23]
Query expansi on techniques for information retrieval: a survey
Hiteshwar Kumar Azad and Akshay Deepak. “Query expansi on techniques for information retrieval: a survey”. In: Information Processing & Manage- ment 56.5 (2019), pp. 1698–1735
2019
-
[24]
Distributed representations of w ords and phrases and their compositionality
Tomas Mikolov et al. “Distributed representations of w ords and phrases and their compositionality”. In: Advances in neural information processing systems. 2013, pp. 3111–3119
2013
-
[25]
Deep Contextualized Word Repres entations
Matthew Peters et al. “Deep Contextualized Word Repres entations”. In: Proceedings of the 2018 Conference of the North American Cha pter of the Association for Computational Linguistics: Human Languag e Technolo- gies, Volume 1 (Long Papers) . New Orleans, Louisiana: Associati...
2018 doi
-
[26]
Universal Sentence Encoder for Engli sh
Daniel Cer et al. “Universal Sentence Encoder for Engli sh”. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Langu age Pro- cessing: System Demonstrations . Brussels, Belgium: Association for Com- putational Linguistics, Nov. 2018, pp. 169–174. doi: 10.1...
2018 doi
-
[27]
Glove: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, and Christopher Ma nning. “Glove: Global Vectors for Word Representation”. In: Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processin g (EMNLP) . Doha, Qatar: Association for Computational Linguistics, O ct. 2...
2014
-
[28]
Universal languag e model fine- tuning for text classification
Jeremy Howard and Sebastian Ruder. “Universal languag e model fine- tuning for text classification”. In: arXiv preprint arXiv:1801.06146 (2018)
2018 arXiv
-
[29]
XLNet: Generalized Autoregressive Pretraining for Lan- guage Understanding
Zhilin Yang et al. “XLNet: Generalized Autoregressive Pretraining for Lan- guage Understanding”. In: arXiv preprint arXiv:1906.08237 (2019)
2019 arXiv
-
[30]
BERT: Pre-training of Deep Bidirec tional Trans- formers for Language Understanding
Jacob Devlin et al. “BERT: Pre-training of Deep Bidirec tional Trans- formers for Language Understanding”. In: Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and...
2019 doi
-
[31]
A comparison of deep learning based query expansion with pseu do-relevance feedback and mutual information
Mohannad ALMasri, Catherine Berrut, and Jean-Pierre C hevallet. “A comparison of deep learning based query expansion with pseu do-relevance feedback and mutual information”. In: European conference on informa- tion retrieval. Springer. 2016, pp. 709–715
2016
-
[32]
Query expansi on using word embeddings
Saar Kuzi, Anna Shtok, and Oren Kurland. “Query expansi on using word embeddings”. In: Proceedings of the 25th ACM international on conference on information and knowledge management . ACM. 2016, pp. 1929–1932
2016
-
[33]
Document language m odels, query models, and risk minimization for information retrieval
John Lafferty and Chengxiang Zhai. “Document language m odels, query models, and risk minimization for information retrieval”. In: Proceedings of the 24th annual international ACM SIGIR conference on Res earch and development in information retrieval . ACM. 2001, pp. 111–119
2001
-
[34]
Notes on the KL-divergence retrieva l formula and Dirichlet prior smoothing
ChengXiang Zhai. “Notes on the KL-divergence retrieva l formula and Dirichlet prior smoothing”. In: Class notes for introduction to text infor- mation systems at the Dept. of Computer Science (2007)
2007
-
[35]
A comparative study of methods for estimating query language models with pseudo feedback
Yuanhua Lv and ChengXiang Zhai. “A comparative study of methods for estimating query language models with pseudo feedback”. In : Proceedings of the 18th ACM conference on Information and knowledge mana gement. ACM. 2009, pp. 1895–1898
2009
-
[36]
Model-based feedba ck in the lan- guage modeling approach to information retrieval
Chengxiang Zhai and John Lafferty. “Model-based feedba ck in the lan- guage modeling approach to information retrieval”. In: Proceedings of the tenth international conference on Information and knowled ge management. ACM. 2001, pp. 403–410
2001
-
[38]
The probability ranking princip le in IR
Stephen E Robertson. “The probability ranking princip le in IR”. In: Jour- nal of documentation 33.4 (1977), pp. 294–304. 16 Ghafourian et al
1977
-
[39]
Estimatio n of statistical translation models based on mutual information for ad hoc in formation retrieval
Maryam Karimzadehgan and ChengXiang Zhai. “Estimatio n of statistical translation models based on mutual information for ad hoc in formation retrieval”. In: Proceedings of the 33rd international ACM SIGIR confer- ence on Research and development in information retrieval . ACM....
2010
-
[1543]
url: https://www.aclweb.org/anthology/D14-1162
doi: 10.3115/v1/D14-1162. url: https://www.aclweb.org/anthology/D14-1162
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.