{"id":"be0a7655-c741-43c4-b95c-fd7df65ecfea","arxiv_id":"2502.03891","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Rewriting repeated queries with terms or keyqueries derived from previously relevant documents improves retrieval effectiveness in an evolving web corpus, beating transformer re-rankers in the LongEval setting.","lead":"Clicks and relevance judgments from past searches can be reused to rewrite repeated queries, even when the previously relevant pages have changed or disappeared. The paper shows such rewrites boost retrieval quality on a French web-search test bed, and can beat costly neural re-rankers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation Table 3 shows gains over BM25 vanish on never-before-seen documents, so the generalization half of the central claim is unsupported.","rationale":"The reader's weakest-assumption analysis already identifies the counterfactual availability of historical documents as the core risk, and Table 3 as the key evidence. My stress-test agrees with that reading and makes it sharper: the empirical support for generalization is essentially absent, so the abstract and conclusion overstate what is demonstrated. However, the reader's CONDITIONAL verdict already captures this; the condition would be to temper the generalization and transformer-comparison claims and provide the stricter ablation. I do not see a reason to move the verdict to REJECT or UNVERDICTED, because the paper is transparent about the counterfactual setup and the code is available, and the Table 2 result, conditional on the assumption, is internally consistent. The main correction is to the strength of the prose, not to the existence of a valid empirical finding under the stated assumptions.","tokens_in":8973,"tokens_out":3711,"duration_ms":42747,"concrete_test":"Re-run the Table 3 ablation with a stricter target set: restrict newly relevant documents to those whose current version differs substantially from the version at feedback time (S3 < 0.8 per Section 4.2), and test whether BM25keyquery significantly outperforms BM25 on this subset using a paired bootstrap with Bonferroni correction at each timestamp. If the improvement is not significant, Section 5's generalization claim should be retracted and the counterfactual assumption should be presented as a hard requirement, not a benign modeling choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: (i) query rewriting with historical relevance feedback improves effectiveness, and (ii) the approach generalizes beyond previously relevant documents, which is why it is said to outperform costly transformer baselines. Part (i) is only demonstrated under the counterfactual assumption that previously relevant documents remain in the corpus and remain relevant. Part (ii) is the load-bearing bridge: Section 5 claims 'the advanced approaches generalize beyond known query-document pairs, making them effective for new documents as well.' Table 3 tests exactly this by removing all documents observed in previous timestamps from runs and qrels. The result is that BM25RF never significantly beats BM25, and BM25keyquery is significant at only 1 of 5 timestamps (01/24, +0.032). Thus the large Table 2 margins (e.g., keyquery nDCG@10' 0.642 vs monoT5 0.405) are almost entirely attributable to the system retrieving the very documents whose relevance labels were used to construct the rewrite. The S3 analysis in Section 4.2 shows roughly 50% of overlapping URLs have similarity below 0.8, so the counterfactual assumption is not innocuous in realistic drift. Without significant gains on genuinely new relevant documents, the generalization claim, and the argument that keyqueries are preferable to simple boosting, are not established by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of using historical relevance feedback for recurring queries in evolving web corpora. It proposes three approaches built on BM25: boosting previously relevant documents by score adjustment (BM25Boost), expanding queries with top tf-idf terms from previously relevant documents (BM25RF), and constructing keyqueries that retrieve previously relevant documents in the top ranks (BM25keyquery). The approaches are evaluated on the LongEval collection at timestamps with overlapping queries, compared against BM25, BM25RM3, ColBERT, List-in-T5, and monoT5, using nDCG@10 and nDCG@10' (with unjudged documents removed). Table 2 reports large gains for the proposed methods. An ablation (Table 3) that removes all previously seen documents from runs and judgments is meant to test generalization to new documents. The paper concludes that rewriting queries with historical relevance feedback improves effectiveness and generalizes beyond known query-document pairs.","tokens_in":9247,"tokens_out":9461,"duration_ms":89919,"significance":"If the claims are valid, the contribution is practically relevant: a lightweight, precomputable query-rewriting strategy could exploit historical clicks or judgments to improve retrieval on recurring queries without retraining expensive neural models. The paper is clearly written, makes code available, and includes significance tests and an ablation that directly targets generalization. However, the current evidence does not support the full strength of the claims. The comparison with transformer baselines is asymmetric because those baselines do not receive the historical relevance signal, and the ablation shows that gains on never-before-seen documents are largely not significant. The realistic significance is therefore narrower than the paper argues: the methods appear to help mainly by retrieving previously relevant documents under the counterfactual assumption that those documents remain relevant, rather than by demonstrating robust generalization to genuinely new documents.","major_comments":[{"comment":"The generalization claim is not supported by the ablation. Table 3 shows that after removing all documents seen in previous timestamps from runs and relevance judgments, BM25RF never achieves a statistically significant improvement over BM25 at any of the five timestamps, and BM25keyquery is significant at only one timestamp (01/24, +0.032). Therefore the statements in Section 4.3 ('indicating that these approaches generalize to newly created or modified documents') and Section 5 ('The ablation study suggests that the advanced approaches generalize beyond known query-document pairs') overstate the evidence. The large margins in Table 2 may be largely attributable to retrieving the very documents whose historical relevance labels were used to construct the rewrites. The paper should either supply additional evidence of generalization (e.g., a more powerful analysis focused on new relevant documents, or per-query breakdowns) or explicitly restrict the contribution to improving retrieval of previously relevant documents.","section":"Section 4.3, Table 3, Section 5"},{"comment":"The comparison with transformer-based baselines is asymmetric. BM25RF and BM25keyquery are constructed using historical relevance judgments (previous qrels), whereas ColBERT, List-in-T5, and monoT5 are run on the current query and corpus only, with no access to the same historical relevance signal. The abstract's claim that the approach 'even outperforms computationally expensive transformer-based approaches' therefore conflates the effect of the extra relevance signal with a difference in retrieval models. The authors should either give the transformer baselines access to historical feedback in a comparable way (e.g., by fine-tuning on previous qrels or using a feedback-based expansion) or reframe the comparison as 'with historical feedback versus without' and soften the claim of general superiority over transformers.","section":"Section 4.1, Table 2"},{"comment":"The sentence 'Keyqueries are the most effective approach in all cases, outperforming the best transformer by a large margin' is not consistent with Table 2. For nDCG@10', BM25RF is higher than BM25keyquery at timestamps 01/23 (0.590 vs 0.574) and 08/23 (0.426 vs 0.422); for nDCG@10, BM25Boost is higher at 07/22, 01/23, and 06/23. Please correct the claim and, if keyqueries are to be recommended as the best variant, report significance tests for the pairwise differences among the proposed approaches.","section":"Section 4.3"}],"minor_comments":[{"comment":"The manuscript contains several typos and small language errors, e.g., 'We proposed' in the Abstract, 'recommendet' and 'whreas' in Section 4.3, 'tocuments' in Section 4.1, 'beyound' in Section 4.3, and 'students t-test' in Table 3's caption. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The timestamps in Figure 1 (06/23, 07/23, 09/23, 01/24, 06/24, 08/24) are inconsistent with Table 2 (07/22, 09/22, 01/23, 06/23, 08/23), and the text says the evaluation runs between June 2022 and August 2023. Please align the labels and clarify the actual evaluation timestamps.","section":"Figure 1 and Table 2"},{"comment":"Table 3 reports only the improvement over BM25; including the BM25 baseline nDCG@10' values would help readers judge the practical magnitude of the deltas and the variability across timestamps.","section":"Table 3"},{"comment":"The paper notes the limitation that previously relevant documents may no longer be available or relevant, but it would be informative to connect this directly to the observed S3 distribution (about 50% of overlapping URLs have similarity below 0.8) and to the weak generalization results in Table 3; currently the connection is not drawn.","section":"Section 4.2 and Section 5"},{"comment":"In the formal definition of keyqueries, the parameter l is not defined; it only appears later as the 'more than 25 results' constraint in Section 4.1. Please define l in the definition.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically clear and reproducible, and the idea of using historical relevance feedback via query rewriting is worth exploring. However, the main claims are currently stronger than the evidence: the generalization result is essentially negative in the ablation, and the transformer comparison is not on equal footing. A revised version that either adds substantial evidence of generalization or re-scopes the contribution to re-finding previously relevant documents, with a corrected comparison, would be within reach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a clear, precomputable method, but the headline comparison against transformers is not apples-to-apples, and the generalization claim is much weaker than the abstract implies. Table 3 is the key: when previously seen documents are removed, BM25RF never significantly beats BM25 and BM25keyquery only once out of five timestamps. The large margins in Table 2 come mostly from retrieving the very documents whose relevance labels built the rewrite. That doesn't kill the paper, but it changes what it is: a demonstration that counterfactual historical feedback helps for re-finding previously relevant documents, not a demonstration that it generalizes to new relevant documents.\n\nWhat's new: the combination of boosting, explicit relevance feedback, and keyqueries used counterfactually against superseded document versions in a time-evolving corpus. That is a reasonable extension of prior keyquery and boosting work, and the authors are upfront about the components. Code is public, methods are specified clearly, and significance tests plus the ablation are included. The S3 analysis on document drift is a nice addition.\n\nSoft spots: the transformer baselines (ColBERT, monoT5, List-in-T5) do not receive the historical relevance signal, so 'outperforms transformers' really means a lightweight system with extra historical labels beats zero-shot transformers without those labels. That is a useful operational finding but not a fair model comparison. The counterfactual assumption that deleted or changed documents remain usable is acknowledged in the text, and the drift analysis shows about half of overlapping URLs have S3 below 0.8, so it is not an innocent assumption. The conclusion says the advanced approaches generalize beyond known query-document pairs based on Table 3, but the statistics do not support that at more than one timestamp. The Matthews effect note in the limitations is honest, but the main text still leans too hard on the generalization claim.\n\nBottom line: the paper deserves a serious referee because it has a reproducible setup, a plausible efficiency story, and a useful negative-ish result buried in the ablation. The authors should be pushed to reframe the claim, rerun the transformer baselines with historical fine-tuning or at least acknowledge the asymmetry explicitly, and soften the generalization language.","headline":"Useful, reproducible counterfactual-feedback study whose generalization claim outruns Table 3; solid paper but needs reframing.","tokens_in":9766,"tokens_out":1569,"would_cite":true,"duration_ms":14874,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using historical relevance feedback to rewrite recurring queries lets a lightweight BM25 retriever beat transformer-based rankers.","keywords":["Query rewriting","Keyqueries","Longitudinal evaluation","Relevance feedback","Temporal information retrieval","Web search","BM25","LongEval"],"falsifier":"Run the same three rewrites on a longitudinal collection in which most previously relevant documents have been deleted and replaced with genuinely new information, and compare nDCG@10 against plain BM25; if the improvement disappears or turns negative while transformer baselines still help, the counterfactual availability of old documents is what drives the reported gains.","tokens_in":8811,"feed_emoji":"🔍","tokens_out":9180,"duration_ms":90180,"temperature":0.7,"pith_summary":"This paper argues that when a search engine sees a query it has seen before, the documents users previously found relevant can be rewritten into a better query even if those documents have since been edited or deleted. The authors call this a counterfactual use of historical relevance feedback: they use the version of the document from the time the relevance judgment was made, as though that version still exists. On the LongEval web-search collection, three BM25-based approaches built on this idea—boosting with old relevance labels, expanding the query with tf-idf terms from old relevant documents, and deriving minimal 'keyqueries' that place old relevant documents at the top of the current corpus—all improve nDCG@10 over the plain BM25 baseline, and the keyquery variant is the most effective. The claim matters because the rewrites can be computed offline, so a lightweight system could serve recurring queries better than expensive transformer-based rankers without running them at query time.","feed_headline":"Old relevance feedback rewrites queries and beats neural rankers","feed_subtitle":"BM25 rewrites from old relevant documents beat ColBERT and monoT5 on recurring web queries without neural inference.","key_machinery":"The central mechanism is the keyquery, a reformulation that is built and then tested against the historical relevant documents before it is used. For a query $q$, the target set $D^+$ contains all documents with positive historical judgments; a keyquery is a minimal query such that every document in $D^+$ appears in the top-k results of the current corpus, the query has more than a threshold number of results, and no proper subquery satisfies both conditions. Candidate keyqueries are constructed from the top ten RM3 terms, and the candidate with the highest nDCG@10 on $D^+$ is selected. The ranking check is what separates this approach from ordinary relevance feedback: it ensures the rewritten query actually positions the previously relevant documents at the top, trading off specificity against the risk of overfitting. A simpler variant used in the paper expands the query with the top ten tf-idf terms from $D^+$ without this verification.","core_discovery":"The paper's central claim is that historical relevance feedback, applied counterfactually to an evolved corpus, can be converted into query rewrites that outperform both ordinary BM25 and transformer-based ranking models. The keyquery is the load-bearing device: for a recurring query $q$ and the set $D^+$ of documents previously judged relevant, a keyquery $q_k$ is a minimal expanded query that retrieves every document in $D^+$ within the top-k results against the current corpus while still returning more than a threshold number of results. Candidate rewrites are generated from the top ten RM3 expansion terms and the best one is chosen by its nDCG@10 on $D^+$. In the LongEval evaluation, this keyquery approach reaches higher nDCG@10 than ColBERT, monoT5, and List-in-T5 on every tested timestamp, with simple tf-idf expansion also beating them on every timestamp.","pith_inferences":["An extension the paper leaves implicit is feeding noisy click logs rather than graded judgments into the same rewrites; if a handful of imperfect clicks suffice, the approach becomes deployable with interaction data alone.","The paper's Matthews-effect caveat suggests a testable boundary: with a time-decay penalty on historical feedback, one could find how quickly the rewrite signal turns from helpful to harmful as document turnover accelerates.","Because the ablation removes previously seen documents and most of the gain disappears, the method may be best understood as a recall booster for old documents; a freshness threshold on the target set would be a natural safeguard before deployment."],"forward_implications":["Search engines that see the same query repeatedly can precompute rewritten queries from accumulated relevance feedback, so each repeat query costs one BM25 lookup while ranking better.","The keyquery variant gives a parameter-light way to keep improving recurring-query results as feedback accumulates, without retraining or GPU inference.","The ablation shows the rewrites can surface documents that were never seen before, where pure boosting of old documents contributes nothing.","In longitudinal setups like LongEval, historical-feedback lexical systems become a reference point that neural rankers without such feedback must be measured against."],"supporting_citations":[{"why":"supplies the LongEval retrieval test collection and its repeated-query timestamps used for all experiments","marker":"[16]"},{"why":"defines the CLEF LongEval lab task that frames the longitudinal evaluation scenario","marker":"[2]"},{"why":"introduces keyqueries, the central reformulation device the paper adapts to historical relevance feedback","marker":"[17]"},{"why":"supplies the keyquery candidate-generation algorithm that the paper re-implements","marker":"[14]"},{"why":"contributes the prior relevance boosting approach and hyperparameter choices that BM25Boost builds on","marker":"[21]"},{"why":"defines BM25, the underlying retrieval model for all three proposed approaches","marker":"[27]"},{"why":"provides the experimentation framework and default RM3 settings used to implement the baselines and rewrites","marker":"[25]"},{"why":"defines ColBERT, one of the transformer baselines the proposed approaches are compared against","marker":"[22]"},{"why":"defines monoT5, the transformer reranker whose effectiveness the keyquery approach is claimed to exceed","marker":"[26]"},{"why":"defines List-in-T5, the listwise reranking baseline used in the comparison","marker":"[31]"}],"fun_headline_variants":["Counterfactual rewrites beat neural rankers on recurring queries","Keyqueries from old relevance feedback outperform ColBERT and monoT5","Query rewrites from historical relevance beat transformer models","Reuse old relevance signals to rewrite queries, outrank neural nets","Historical feedback rewrites queries; keyqueries top neural ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the counterfactual assumption that a document judged relevant at an earlier time remains a usable signal even if it has been edited or deleted; if relevance drifts quickly, the rewrites would push rankings toward outdated content.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual rewrites beat neural rankers on recurring queries","Keyqueries from old relevance feedback outperform ColBERT and monoT5","Query rewrites from historical relevance beat transformer models","Reuse old relevance signals to rewrite queries, outrank neural nets","Historical feedback rewrites queries; keyqueries top neural ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1266,"prompt_tokens":862,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":478,"tokens_out":404,"duration_ms":4742,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:20:47.210217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three rewrites on a longitudinal collection in which most previously relevant documents have been deleted and replaced with genuinely new information, and compare nDCG@10 against plain BM25; if the improvement disappears or turns negative while transformer baselines still help, the counterfactual availability of old documents is what drives the reported gains.","supporting_citations":[{"cited_title":"In: Chen, H., Duh, W.E., Huang, H., Kato, M.P., Mothe, J., Poblete, B","cited_arxiv_id":null,"evidence_quote":"supplies the LongEval retrieval test collection and its repeated-query timestamps used for all experiments"},{"cited_title":"In: CLEF (Working Notes)","cited_arxiv_id":null,"evidence_quote":"defines the CLEF LongEval lab task that frames the longitudinal evaluation scenario"},{"cited_title":"In: Gurrin, C., Jones, G., Kelly, D., Kruschwitz, U., de Rijke, M., Sakai, T., Sheridan, P","cited_arxiv_id":null,"evidence_quote":"introduces keyqueries, the central reformulation device the paper adapts to historical relevance feedback"},{"cited_title":"In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology","cited_arxiv_id":null,"evidence_quote":"supplies the keyquery candidate-generation algorithm that the paper re-implements"},{"cited_title":"In: Faggioli, G., Ferro, N., Galuscáková, P., de Herrera, A.G.S","cited_arxiv_id":null,"evidence_quote":"contributes the prior relevance boosting approach and hyperparameter choices that BM25Boost builds on"},{"cited_title":"(Jan 1994)","cited_arxiv_id":null,"evidence_quote":"defines BM25, the underlying retrieval model for all three proposed approaches"},{"cited_title":"In: Balog, K., Setty, V., Lioma, C., Liu, Y., Zhang, M., Berberich, K","cited_arxiv_id":null,"evidence_quote":"provides the experimentation framework and default RM3 settings used to implement the baselines and rewrites"},{"cited_title":"In: Huang, J.X., Chang, Y., Cheng, X., Kamps, J., Murdock, V., Wen, J., Liu, Y","cited_arxiv_id":null,"evidence_quote":"defines ColBERT, one of the transformer baselines the proposed approaches are compared against"}],"review_version":1}