{"id":"f029f196-4f66-4eb3-9acf-d0bb42078fb5","arxiv_id":"2411.15530","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A selective, ELMo-based question expansion method yields a small but statistically significant MAP gain in CQA question retrieval.","lead":"Researchers compared query expansion methods for finding similar questions in community Q&A archives, and proposed adding an ELMo-based similar-question expansion plus selective exclusion of key words. Their best variant improved retrieval accuracy (MAP) by 1.8% relative to the strongest baseline without query expansion.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.7428 MAP hinges on the under-specified centrality selector in §3.3: Eq. 16's free parameter c is unreported and the two-central-word intersection can be empty, so the headline result cannot be independently reproduced or sensitivity-checked.","rationale":"The reader's weakest_assumption correctly zeroes in on the Paik-Oard centrality component. I sharpen this into a concrete, load-bearing objection: the paper not only omits the centrality algorithm's details but also leaves a free parameter (c in Eq. 16) and an undefined edge case (empty intersections for two-central-word questions). These are not mere stylistic omissions; they change the method's output. The central claim's key number, 0.7428, is produced by a method whose core selective step cannot be reconstructed from the paper. This is more damaging than a pure 'transfer from verbose queries' concern because it blocks verification entirely: an independent reimplementation cannot match the reported result without additional information, and the sensitivity of the result to c is unknown. The significance test is also under-reported, but if the method cannot be reproduced, the significance claim is moot. The proposed concrete test directly settles whether the concern lands by varying c and checking the empty-intersection frequency. If the MAP remains stable across c and empty intersections are rare or handled, the concern fails and the paper's method is likely sound. If not, the headline result is fragile. My verdict remains CONDITIONAL because the issues are addressable in a revision with full specification of the centrality algorithm, the c value, and the empty-intersection behavior.","tokens_in":11008,"tokens_out":8214,"duration_ms":75590,"concrete_test":"Re-implement the centrality algorithm from Paik & Oard (CIKM 2014), set c in Eq. 16 to 0.1, 1, 10, and 100, and re-run exp-ELMoPRF-centrality on the same 630-query test split. Also log the fraction of two-central-word questions whose expansion-set intersection is empty. If the resulting MAP falls below the significance threshold relative to TR-LM (0.7291) for any c, or if more than 2% of queries are silently excluded from ELMo expansion due to empty intersections, the headline result is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a statistically significant 1.8% relative MAP gain of exp-ELMoPRF-centrality over TR-LM (0.7428 vs 0.7291). This method's selective-expansion component relies on the Paik-Oard centrality algorithm [37], but §3.3 explicitly omits its details ('we omit the full details of the algorithm') and Eq. 16 contains an unreported free parameter c. Since the centrality vector A determines which words are excluded from the ELMo similarity computation (Eqs. 18-19), any implementation choice for c changes the expansion set. Moreover, for questions with two central words (Table 2), the final feedback set is the intersection of two k-nearest-neighbor sets; the paper never states how an empty intersection is handled, leaving the method's behavior undefined for a subset of the 630 test queries. Without these details, the reported 0.7428 cannot be reproduced or checked for sensitivity, and the claimed significance cannot be audited.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the lexical gap problem in community question answering (CQA) question retrieval by exploring query expansion methods. It adapts two existing word-embedding-based expansion methods (expAL and expKuzi), proposes a question-similarity-based expansion method using ELMo contextualized representations (expELMo), and augments it with pseudo-relevance feedback (expELMoPRF). A further contribution is selective expansion, which identifies 'central' terms using the Paik-Oard centrality method and excludes them from the expansion process, yielding expAL-centrality, expKuzi-centrality, expELMo-centrality, and exp-ELMoPRF-centrality. On the Zhang et al. test collection, exp-ELMoPRF-centrality reports MAP 0.7428, a claimed statistically significant 1.8% relative improvement over the best non-expansion baseline TR-LM (MAP 0.7291). The paper also claims that ELMo-selected similar questions provide better feedback than PRF-selected questions.","tokens_in":11173,"tokens_out":5779,"duration_ms":50470,"significance":"If the best method is fully specified and reproducible, this is a useful empirical exploration: it systematically compares several expansion strategies on a public relevance-judged collection and provides evidence that excluding central words during expansion can improve retrieval. A strength is the use of an external test collection and a held-out development split for parameter tuning. The headline improvement is modest (1.8% relative over TR-LM), so the paper's value depends on the reliability of the significance claim and on the completeness of the method description. The under-specification of the centrality component currently prevents independent reproduction, which is the main obstacle to accepting the empirical claims.","major_comments":[{"comment":"The centrality algorithm is explicitly omitted ('we omit the full details of the algorithm'), and the value of the IDF regularization constant c in Eq. (16) is never reported nor shown to be tuned in Section 4.2. Because the final term importance I(qi) = A(qi) * didf(qi) determines which terms are designated central and hence excluded from expansion in the best-performing method, the headline result (exp-ELMoPRF-centrality, Table 5) cannot be independently reproduced or sensitivity-checked. Please provide the full algorithm (or a precise pointer to the exact equations in Paik and Oard [37]) and report the value of c; if c was fixed without tuning, please provide a sensitivity analysis over a range of c values.","section":"Section 3.3, Eq. (16)"},{"comment":"For questions with two central words, the final ELMo feedback set is specified as the intersection of two k-nearest-neighbor sets obtained after excluding each central word in turn. The paper does not state how an empty intersection, or an intersection smaller than k, is handled. Since Table 2 explicitly shows two-central-word cases, the expansion procedure is undefined for a potentially non-negligible subset of the 630 test queries. Please specify the fallback rule (e.g., union, use only one of the two sets, or reduce k) and, if possible, report how many test queries have two central words and how many of those yield empty or sub-k intersections.","section":"Section 3.3, Eq. (19)"},{"comment":"The claim that all improvements in Table 5 are 'statistically significant ... using t-test with 95% confidence level' is not accompanied by the test variant (paired or unpaired), the test statistic, or the p-value. With only 630 test queries and a 1.8% relative gain, the significance claim is load-bearing for the paper's main conclusion. Please report the details of the test, including the unit of analysis (e.g., per-query average precision), degrees of freedom, and the exact p-value (or a confidence interval) for exp-ELMoPRF-centrality versus TR-LM, and ideally for the other pairwise comparisons claimed to be significant.","section":"Section 4.3, Tables 4 and 5"}],"minor_comments":[{"comment":"The KL-divergence scoring formula appears garbled: 'log pseen (w|θ QC ) α dp (w|C)' seems to be missing the fraction bar and the denominator's α_d term. Please rewrite the equation cleanly so that the smoothing notation is unambiguous.","section":"Section 3, Eq. (1)"},{"comment":"The text groups '[24] and [27]' as 'word2vec models'; reference [27] is Pennington et al.'s GloVe, which is a different count-based embedding model. Please correct the wording to 'word embedding models' or otherwise distinguish the two.","section":"Section 2.3"},{"comment":"There is a typo 'euq. 7' in the sentence before Eq. (18); it should be 'Eq. (7)'. Also, the name 'paik et al.' should be capitalized as 'Paik et al.' for consistency.","section":"Section 3.3"},{"comment":"The table caption does not explain the blue and red color coding used to distinguish the central word before and after applying IDF. Since the paper may be read in grayscale, please add textual markers (e.g., asterisks or superscripts) to the table.","section":"Table 2"},{"comment":"The text says α values in expAL were tested in 'period [0.2, 0.4]'; 'period' should be 'range'. Also, please clarify whether the α and β values in expELMoPRF-centrality were tuned jointly or independently, and over what ranges.","section":"Section 4.2"},{"comment":"The claim that expELMo outperforms LM-PRF ('we can conclude that using questions ... are more accurate compared to the questions that are suggested by LM-PRF') is based on MAP 0.7375 versus 0.7328 without a reported significance test. Either add a significance test for this comparison or soften the conclusion.","section":"Section 4.3"},{"comment":"The paper does not state whether all baselines and proposed methods use the same preprocessing (e.g., stopword removal, stemming, Dirichlet prior μ). Please report these settings so that the comparisons are fully interpretable.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable empirical exploration, but the reproducibility gap in Section 3.3 is the main obstacle to accepting the reported results. The modest 1.8% relative gain and the small 630-query test split make the significance claim particularly sensitive to the exact implementation choices. I recommend requiring a complete specification of the centrality algorithm and the missing parameter c, plus a clear statement of the intersection fallback, before publication. The paper's scope seems better suited to a short conference paper or a workshop paper, but the experiments are adequately designed apart from the omissions noted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clean empirical study of query expansion for CQA question retrieval. The genuinely new pieces are the ELMo-based similar-question feedback (expELMo) and the centrality-based selective expansion; the best method, exp-ELMoPRF-centrality, reports a 1.8% relative MAP gain over TR-LM (0.7428 vs 0.7291). That is a small but real improvement if it holds.\n\nWhat it does well: uses a held-out dev/test split from the Zhang et al. collection, compares against four baselines including a strong LM-PRF, and reports significance for the main result. The evaluation design is honest, and the gains are consistent across variants: every selective-expansion variant beats its non-selective counterpart, which supports the intuition that not expanding key words helps.\n\nThe load-bearing problem is reproducibility. Section 3.3 explicitly omits the Paik-Oard centrality algorithm and never reports the free parameter c in Eq. 16. Since the centrality vector determines which words are excluded from expansion, a different c or a different implementation could erase the 1.8% gain. Worse, for questions with two central words, the final feedback set is the intersection of two k-NN sets, and the paper never says what happens when that intersection is empty. These are not philosophical objections; they make the headline result impossible to reproduce or sensitivity-check from the text alone. The stress-test note is correct on both counts.\n\nA second issue: the paper claims ELMo feedback beats PRF feedback based on expELMo (0.7375) vs LM-PRF (0.7328), but that comparison is not significance-tested; only the central-variant table gets the t-test footnote. Third, ten hyperparameters are tuned on a 630-query dev set, and the reported gains are on the order of 1-2%, so selection effects are a legitimate concern. These are minor-to-moderate; they do not sink the paper but they should be fixed.\n\nWho it's for: people working on CQA retrieval or lexical gap. They'll get a useful recipe and a cautionary example of evaluation. It deserves a serious referee — the method is novel and the eval is careful — but the referee should ask for the missing details or the code.\n\nSend it to peer review, with a revision contingent on full disclosure of the centrality algorithm, the value of c, and the empty-intersection handling.","headline":"A careful, modest empirical study whose headline 1.8% gain is plausible but currently impossible to verify because the centrality-based expansion is under-specified.","tokens_in":11814,"tokens_out":2832,"would_cite":false,"duration_ms":22536,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68P20","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that expanding a user's question with words from the most semantically similar questions, selected by a contextualized language model and with the question's central words withheld from expansion, raises retrieval…","keywords":["community question answering","question retrieval","query expansion","ELMo","pseudo-relevance feedback","selective expansion","lexical gap","word embedding"],"falsifier":"Re-running the experiment with the centrality regularizer constant c swept over a grid, or evaluating on a held-out split larger than the 630 test questions used, would show whether the 0.7428 MAP holds or regresses toward the LM-PRF level of 0.7328.","tokens_in":10721,"feed_emoji":"🔎","tokens_out":6810,"duration_ms":54391,"temperature":0.7,"pith_summary":"This paper tries to show that the lexical gap in community question answering can be narrowed by expanding the user's question before retrieval. Its core move is to expand not with related words but with words from the most semantically similar questions, found by ELMo contextualized embeddings, and to exclude a small set of \"central\" words whose expansion would change the question's intent. The best configuration, exp-ELMoPRF-centrality, reaches Mean Average Precision 0.7428 on a 1,260-question test set, about 1.8% relative better than the strongest non-expansion baseline (TR-LM). If this holds, question retrieval gains a modest but reliable improvement at retrieval time. The paper also argues that ELMo-selected similar questions serve as better pseudo-feedback than PRF-retrieved questions.","feed_headline":"ELMo question expansion lifts retrieval MAP by 1.8 percent","feed_subtitle":"Expanding with similar questions while shielding central words pushes community-Q&A retrieval MAP to 0.7428.","key_machinery":"The machinery is an expanded question language model built by interpolating three components: the original question's maximum-likelihood language model, a language model estimated from the k most ELMo-similar questions (k=5), and a pseudo-relevance feedback language model over the first two retrieved questions (weights 0.3, 0.3, and 0.2 respectively). ELMo, a pre-trained bidirectional language model, supplies context-dependent word vectors that are length-scale averaged to represent each question; cosine similarity between these vectors selects the expansion questions. A fixed-point term-centrality algorithm marks one or two words as the question's central intent-bearing words, and those words are removed from the expansion process: their vectors are excluded from question representations, and for two-word cases the expansion sets are intersected. This selectivity is what turns the word- and question-similarity expansions into consistently significant gains.","core_discovery":"The central claim is that expanding an input question with terms drawn from the most semantically similar questions, selected using ELMo-based contextualized representations and with the question's central words excluded from expansion, improves question retrieval performance on the 1,260-question test set to a MAP of 0.7428, a statistically significant relative improvement of about 1.8% over the TR-LM baseline (0.7291). The same experiments support a second claim: feedback questions chosen by ELMo similarity are more useful for expansion than feedback questions chosen by pseudo-relevance feedback alone, because ELMo captures semantic and syntactic similarity rather than just lexical overlap.","pith_inferences":["A natural next test the paper leaves implicit is whether the same selective-expansion idea carries over to transformer-based dense retrievers; the gains here are demonstrated on lexical matching models, so the margin may shrink or grow under a different retrieval backbone.","The paper's own Table 2 suggests a testable refinement: when two words jointly carry a question's intent, expanding either one alone may distort the intent less than expanding both, and dropping both may be safest; this could be evaluated by a controlled ablation.","Because the centrality algorithm's IDF regularizer constant c is never reported, a reproducibility check should fix c on a development set before trusting the selective-expansion component; this omission makes the selectivity contribution partially under-specified as published."],"forward_implications":["Question retrieval systems that already use KL-divergence language models could adopt this expansion recipe without retraining, since the gains come from retrieval-time interpolation rather than learned rankers.","The finding that ELMo-selected questions outperform PRF-selected questions as feedback suggests that building feedback sets from semantic similarity rather than first-pass lexical retrieval is a sufficient condition for the improvement.","Withholding central words from expansion is shown to help across all three expansion families (word-level, whole-question, and question-level), indicating that selective expansion is transferable beyond the specific ELMo configuration.","The reported 1.8% relative MAP gain is small but consistent, and it is achieved on top of already strong language-model and translation-model baselines, meaning the method addresses residual lexical gap rather than a weak baseline."],"supporting_citations":[{"why":"It supplies ELMo, the pre-trained contextualized language model whose pooled vectors represent questions for the similarity-based expansion.","marker":"[25]"},{"why":"It provides the word-by-word expansion method (expAL) that the paper adapts and later improves with selective expansion.","marker":"[31]"},{"why":"It provides the whole-question word-embedding expansion method (expKuzi), the other word-similarity competitor and the source of the interpolation formula.","marker":"[32]"},{"why":"It supplies the fixed-point term-centrality method used to identify central words and to justify leaving those words out of the expansion.","marker":"[37]"},{"why":"It supplies the test collection of 1,260 input questions with labeled relevant questions on which all methods are evaluated.","marker":"[16]"},{"why":"It provides the translation-based language model (TR-LM) that is the strongest non-expansion baseline and the reference for the claimed 1.8% improvement.","marker":"[3]"},{"why":"It supplies the simple mixture model used to estimate the pseudo-relevance feedback language model in the PRF and ELMo-PRF expansions.","marker":"[36]"}],"fun_headline_variants":["ELMo question similarity lifts CQA retrieval MAP by 1.8%","1.8% MAP gain from ELMo-expanded queries in CQA","ELMo-based expansion boosts question retrieval 1.8%","Excluding central words helps ELMo expansion gain 1.8%","CQA retrieval MAP up 1.8% with ELMo-selected expansion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage depends on the algorithm that picks a question's central words behaving on short question titles the same way it behaves on the long web queries it was designed for; the paper omits the algorithm's full details and never reports a key internal constant, so if that transfer fails the gain could disappear.","fun_headline_variants_meta":{"raw":{"variants":["ELMo question similarity lifts CQA retrieval MAP by 1.8%","1.8% MAP gain from ELMo-expanded queries in CQA","ELMo-based expansion boosts question retrieval 1.8%","Excluding central words helps ELMo expansion gain 1.8%","CQA retrieval MAP up 1.8% with ELMo-selected expansion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000711,"raw_usage":{"total_tokens":3120,"prompt_tokens":783,"completion_tokens":2337,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":2253}},"tokens_in":399,"tokens_out":2337,"duration_ms":14673,"temperature":1.0,"reasoning_tokens":2253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:11:19.111917+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the experiment with the centrality regularizer constant c swept over a grid, or evaluating on a held-out split larger than the 630 test questions used, would show whether the 0.7428 MAP holds or regresses toward the LM-PRF level of 0.7328.","supporting_citations":[{"cited_title":"A comparison of deep learning based query expansion with pseu do-relevance feedback and mutual information","cited_arxiv_id":null,"evidence_quote":"It provides the word-by-word expansion method (expAL) that the paper adapts and later improves with selective expansion."},{"cited_title":"Query expansi on using word embeddings","cited_arxiv_id":null,"evidence_quote":"It provides the whole-question word-embedding expansion method (expKuzi), the other word-similarity competitor and the source of the interpolation formula."},{"cited_title":"A fixed-point method for weighting terms in verbose informational queries","cited_arxiv_id":null,"evidence_quote":"It supplies the fixed-point term-centrality method used to identify central words and to justify leaving those words out of the expansion."},{"cited_title":"Learning distributed representation s of data in com- munity question answering for question retrieval","cited_arxiv_id":null,"evidence_quote":"It supplies the test collection of 1,260 input questions with labeled relevant questions on which all methods are evaluated."},{"cited_title":"Retrieval models for question and answer archives","cited_arxiv_id":null,"evidence_quote":"It provides the translation-based language model (TR-LM) that is the strongest non-expansion baseline and the reference for the claimed 1.8% improvement."},{"cited_title":"Model-based feedba ck in the lan- guage modeling approach to information retrieval","cited_arxiv_id":null,"evidence_quote":"It supplies the simple mixture model used to estimate the pseudo-relevance feedback language model in the PRF and ELMo-PRF expansions."}],"review_version":1}