{"id":"6784b6ab-8b5d-4e79-9a67-c57a4071dd12","arxiv_id":"2412.07998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fusing BM25 rankings from non-personalized, expanded, and personalized query rewrites achieved the best passage retrieval scores for RALI at TREC iKAT 2024, though no ablation isolates the fusion effect.","lead":"A team from Université de Montréal merged search rankings from three differently-personalized versions of each conversational query to fight 'over-personalization'. Their best fused system topped their TREC iKAT 2024 submissions, but the paper's own analysis suggests the test collection itself is biased.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fusion's effectiveness is not proven: Table 1 lacks a non-fusion automatic control, so the reported gain over manual runs cannot be attributed to retrieval fusion.","rationale":"The reader's weakest assumption focuses on pooling bias, but the more immediate threat to the central claim is the missing control: no automatic run without fusion is reported. Even if the judgments were perfectly unbiased, Table 1 would not show whether fusion, rather than the generated queries or the reranker, caused the improvement. The pooling-bias discussion in Section 5 is candid and relevant, but it does not address the confound. I therefore agree with the conditional verdict but for a reason that is partially different from the reader's stated weakest assumption. The proposed ablation would settle the attribution question directly and is feasible with the released iKAT-24 qrels.","tokens_in":9799,"tokens_out":4655,"duration_ms":47868,"concrete_test":"On the iKAT-24 topics, run the same BM25+monoT5 pipeline using only (a) the non-personalized rewrite, (b) the response-expanded query, and (c) the personalized rewrite, and (d) the full three-list score fusion with the same iKAT-23-tuned weights. Compare NDCG@10 after ensuring (a)-(c) have their top documents judged (e.g., via the official qrels or a fresh pooling round). If (d) does not beat the best single-query variant by a margin larger than the noise from shallow pooling, the central fusion claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4 that the fusion+monoT5 run 'proves the effectiveness of retrieval-fusion' is not supported by the experiments reported in Table 1. The table contains no automatic no-fusion baseline: the two automatic runs both use three-query fusion, and the manual runs use human rewrites, so they differ in query source, number of queries, and reranking input. The observed NDCG@10 gain (47.7 vs 35.4 for manual BM25+monoT5) could come from the automatically generated queries, the response-expanded query, or the monoT5 reranker rather than from fusion. Section 3.2.1 says the fusion weights were tuned on iKAT-23 but does not report them, and no ablation isolates the contribution of the personalized list or the fusion operation. Section 5's assessment-bias discussion does not remedy this: because the fusion run was submitted and any unfused automatic variant was not, the unfused variant's documents would be under-assessed, biasing a comparison in favor of fusion. Thus the paper establishes that one pipeline scores well, not that fusion causes the improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports RALI's participation in the TREC iKAT 2024 passage-ranking task. It identifies an 'over-personalization' problem in which adding PTKB-derived terms to an LLM-generated query rewrite can drift the query, and proposes to mitigate this by retrieving separately with three query variants: a non-personalized LLM rewrite, that rewrite expanded with a generated response, and a personalized LLM rewrite. The three BM25 ranking lists are then fused with a weighted linear combination whose weights are tuned on iKAT-23, and the fused top 50 are optionally reranked with monoT5 using the personalized rewrite as the query. Two manual runs use the provided human rewrites with BM25 plus monoT5 or RankLlama reranking. Official iKAT-24 scores are reported in Table 1; the best run (BM25 fusion + monoT5) reaches NDCG@10 47.7 versus 35.4 for manual BM25 + monoT5. The paper concludes that retrieval fusion is effective and discusses assessment bias in shallow-pooled iKAT collections, including a single-turn case study in which an expanded human rewrite has fewer assessed documents in the top 20.","tokens_in":10160,"tokens_out":9879,"duration_ms":89899,"significance":"Retrieval fusion is a plausible and inexpensive hedge against over-personalization, and the iKAT-24 evaluation is, with respect to the tuned weights, an out-of-collection test. The pipeline is described clearly enough to reproduce its general design, and the assessment-bias discussion raises a genuine methodological concern for future iKAT evaluations. These are real strengths. However, the central claim that fusion, rather than the automatic query variants or the reranking configuration, is responsible for the observed improvement is not established by the reported experiments. The paper's own Section 5 identifies a bias that also affects any non-submitted comparison condition, so the headline conclusion is currently under-determined. An ablation isolating the fusion operation, a report of the tuned weights, and a more systematic treatment of judgment incompleteness would materially strengthen the contribution.","major_comments":[{"comment":"The Section 4 claim that the best run provides 'proving the effectiveness of retrieval-fusion' is not supported by Table 1. The table contains no automatic no-fusion control: both automatic runs use the same three-query fusion, while both manual runs use a single human rewrite. The automatic/manual comparison therefore varies the rewrite source, the number of query variants (including a response-expanded list), and the reranking query (the personalized rewrite for the fused run versus the human rewrite for the manual runs). The NDCG@10 gap between BM25 fusion + monoT5 (47.7) and manual BM25 + monoT5 (35.4) could be produced by any of these factors alone. Please add an ablation with an unfused automatic single-list run, or equivalently remove one list at a time, and restrict the wording to what Table 1 can show.","section":"§4, Table 1"},{"comment":"The fusion weights alpha_1, alpha_2, and alpha_3 are said to be 'optimized based on performance tuning on the iKAT-23 test collection,' but their values are never reported and no sensitivity analysis is given. Without the weights, the linear-combination mechanism cannot be reproduced or interpreted. In addition, Section 4 states that on iKAT-23 the fusion-then-reranking approach did not outperform manual-rewrite runs; because iKAT-23 is the same collection used for weight tuning, that result is in-sample and cannot be used to validate the method. Please report the weights and, ideally, performance across a range of weight settings.","section":"§3.2.1, Eq. (1)"},{"comment":"The assessment-bias discussion draws on one illustrative turn (9-1-3) and on collection-level pooling statistics, which is useful context but not sufficient to support a causal reading of Table 1. The paper does not provide per-condition judged-document statistics for the actual iKAT-24 runs, and no unfused automatic variant was submitted; consequently, any post-hoc ablation on iKAT-24 would be subject to the same bias the paper documents, because documents retrieved only by the unfused variant would be less likely to have been judged. A systematic comparison of assessed-document rates in the top ranks of each condition, or re-judging of a sample of documents, is needed before the observed scores can be attributed to fusion.","section":"§5, Tables 2-3"},{"comment":"The fusion formula is a weighted sum of BM25 scores, but the paper does not state whether scores are normalized before combination. BM25 score scales differ across queries and query types, so a raw weighted sum can be dominated by whichever list happens to have larger score magnitudes, which would undermine the intended 'chorus effect.' If normalization is applied, it should be described; if not, this is a technical gap in the proposed mechanism. The defaulting of missing documents to the score of the 1000th document also needs to be reconciled with whatever normalization is used.","section":"§3.2.1, Eq. (1)"}],"minor_comments":[{"comment":"The manuscript contains typographical errors, including 'Retrie val Fusion' in the title, 'and hinders search performance T his' in the abstract, and 'Rermarkably' in Section 2; it should be proofread.","section":"Title and abstract"},{"comment":"The text says four automatic runs were submitted but only two are described or listed; either include the other two runs in Table 1 or state clearly why they are omitted.","section":"§3.2"},{"comment":"The fusion equation is not numbered; numbering it would make the technical discussion and any future citations easier.","section":"§3.2.1"},{"comment":"Table 1 would be more informative with official run identifiers and a paired significance test, or at least a per-topic variance estimate, for the main NDCG@10 comparison.","section":"Table 1"},{"comment":"The personalized rewrite prompt in Section 3.2.1 depends on the LLM4CS framework [20] and on reference [28], but [28] is not discussed in Related Work; a sentence or two describing that prior method would help readers situate the contribution.","section":"§3.2.1, Related Work"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection. The underlying idea is testable and the reported iKAT-24 scores are potentially useful, but the central 'fusion works' claim lacks the necessary control, the fusion weights are unreported, and the assessment-bias discussion needs to be made systematic before it can do the explanatory work assigned to it. These issues are fixable with additional experiments and reporting, so the manuscript is within the scope of a major revision. I would also ask the editor to consider whether the TREC system-report framing matches the journal's expectations for causal claims; the manuscript's conclusions should be moderated accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a TREC iKAT system description, five pages, and it should be read in that frame. The new idea is sound: instead of stuffing every user-profile element into one rewritten query, they generate three query rewrites at different personalization levels, run BM25 on each, and fuse the lists with a weighted linear combination. Applying the old Chorus Effect to LLM-generated personalized and non-personalized rewrites is a legitimate new application of retrieval fusion, and the over-personalization framing is clear and useful.\n\nThe paper has real strengths. The pipeline is described concretely: prompts, rerankers, fusion weights (though the values are not reported). Section 5 is genuinely candid about assessment bias. They show with a concrete example that their own expanded queries retrieve documents that were never judged, and they draw on the pooling literature to question iKAT reusability. That honesty is worth credit.\n\nNow the soft spots. The headline claim in Section 4 says the fusion+monoT5 run proves the effectiveness of retrieval fusion. It does not. Table 1 has no unfused automatic baseline. The two automatic runs both use three-query fusion; the manual runs use a single human rewrite. So the automatic runs differ from the manual runs on query source, query count, response expansion, and reranking input. The observed NDCG gain could come from the LLM rewrites, the expanded query, or monoT5, not from the fusion operation. The weights were tuned on iKAT-23 and not disclosed, which makes the iKAT-23 evaluation in Section 4 in-sample and hard to interpret. The assessment-bias discussion does not rescue the claim; it actually makes an unfused comparison harder, because the unfused variant was not submitted to pooling and would be under-assessed.\n\nThere is a good point in Section 5 that a submitted method cannot be inflated by pooling bias, and that point is fair for the fusion run itself. But that is a claim about the pipeline scoring well, not about fusion being the cause. The citation pattern looks fine; the self-citations are to their own LLM4CS framework, which they actually use as a component, and the fusion literature is properly credited.\n\nThis paper is for people working on conversational search personalization and on the reuse of TREC iKAT collections. The assessment-bias discussion is the most valuable part. The fusion claim is preliminary and needs an ablation: one automatic run with a single non-personalized query, one with a single personalized query, and one fused, with weights reported and ideally evaluated on deeper pools. If that gets added, this becomes a solid short paper. I would send it to peer review as a TREC system description, with the expectation that the fusion claim be softened until the ablation exists.","headline":"A candid TREC system description with a sensible fusion idea, but the central claim that fusion works is not proven by the experiments as reported.","tokens_in":10560,"tokens_out":2370,"would_cite":true,"duration_ms":26379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing personalized and neutral queries improves search ranking.","keywords":["conversational search","personalized query reformulation","retrieval fusion","over-personalization","BM25","assessment bias","iKAT"],"falsifier":"Judge every unassessed document in the top 10 of the fused run for all turns, recompute NDCG@10, and compare with the same recomputation for the best non-fusion run; if the fused run no longer leads, the central claim is falsified. The authors' 9-1-3 example (5 of top 20 judged for the expanded query versus 14 for the original) shows exactly the kind of incompleteness that would matter.","tokens_in":9607,"feed_emoji":"🔎","tokens_out":13434,"duration_ms":118011,"temperature":0.7,"pith_summary":"Personalized conversational search faces a dilemma: user-profile terms can sharpen a query, but too many of them pull retrieval away from the user's true intent. This paper claims that the right response is not to pick a single query rewrite but to fuse ranking lists produced by rewrites at several personalization levels. The submitted best run combines BM25 lists from a non-personalized rewrite, a response-expanded rewrite, and a personalized rewrite using a weighted score sum, then reranks the top documents with a neural reranker, and it outperforms the team's other runs. The paper takes this as evidence that agreement between personalized and non-personalized retrieval, the 'Chorus Effect', identifies documents worth ranking highly. It also argues that the test collections' shallow pooling biases evaluation against non-participant query formulations, so these gains may actually be understated.","feed_headline":"Fusing personalized and neutral queries improves search ranking","feed_subtitle":"A score-based blend of query rewrites at different personalization levels beats the team's other retrieval runs.","key_machinery":"The load-bearing object is a score-based linear-combination fusion of three BM25 ranking lists. For a document $D$, the fused score is $score(D) = \\alpha_1 s_1(D) + \\alpha_2 s_2(D) + \\alpha_3 s_3(D)$, with fixed weights tuned on the previous year's test collection; documents missing from a list receive the score of the 1000th document in that list. This implements the 'Chorus Effect' principle, that when multiple retrieval strategies agree that a document is relevant, the agreement is treated as stronger evidence of relevance, and the top 50 documents of the fused list are then reranked by a neural reranker before evaluation.","core_discovery":"The paper's central claim is that score-based fusion of ranking lists generated from queries with different levels of personalization improves passage ranking in personalized conversational search. The mechanism is a weighted linear combination of three BM25 lists—one from a de-contextualized non-personalized rewrite, one from that rewrite expanded with a generated response, and one from a personalized rewrite that incorporates user-profile elements—followed by reranking the top 50 fused documents with a pretrained sequence-to-sequence reranker. The run with fusion and reranking achieved the strongest scores among the four submitted runs, which the authors present as evidence that retrieval fusion is effective. On the previous year's collection, however, the same fusion-then-reranking approach did not beat manual human rewrites; the paper argues this discrepancy is largely due to assessment bias, since documents retrieved by non-participant query formulations receive far fewer relevance judgments.","pith_inferences":["Beyond the paper, the same fusion principle should apply to other query variants and other retrievers: what matters is that the candidate lists are diverse enough that agreement between them is informative, not the specific BM25 base.","A direct test of the assessment-bias argument would be to have the top unassessed documents from the fused run judged and to recompute the metrics; the paper's own case study predicts the fusion advantage would grow, not shrink, under complete judgment.","The fixed-weight design invites an adaptive extension: estimating how noisy a given turn's personalized rewrite is, and shifting weight toward the non-personalized list when profile terms look risky, could widen the gap over single-query baselines.","For benchmark builders, the paper's bias analysis implies that shallow-pooled conversational collections should not be treated as reusable for methods that did not contribute to the pool, unless supplemental judgments are collected."],"forward_implications":["A conversational search system does not have to choose between a personalized and a non-personalized query; keeping both in the candidate mix can protect against over-personalization.","Score-based fusion with fixed per-query-type weights is a simple, parameter-light strategy that can be tuned on an earlier test collection and applied to new turns without per-turn weight fitting.","Documents that rank highly under both personalized and non-personalized formulations are the high-confidence candidates, so fusion can also be used as a filtering signal before expensive reranking.","Reranking the fused list adds a further gain over fusion alone, so the two stages are complementary in the submitted pipeline.","Because shallow pooling under-assesses non-participant query formulations, a fused run that beats participant runs on such a collection is a conservative signal of its true relative quality."],"supporting_citations":[{"why":"Defines the track and supplies the pool-depth statistics used in the assessment-bias comparison.","marker":"[1]"},{"why":"Shows that large collections with shallow pooling can produce biased relevance sets.","marker":"[4]"},{"why":"Provides the early equal-weight score-combination methods that frame the fusion design.","marker":"[10]"},{"why":"Supplies the prompt-based rewriting approach used to generate the non-personalized and response-expanded queries.","marker":"[20]"},{"why":"Demonstrates that LLMs can build de-contextualized personalized queries and notes the lack of public training data for weight assignment.","marker":"[28]"},{"why":"Provides the pretrained reranker used after fusion and the earlier observation about low judgment rates for non-BM25 documents.","marker":"[30]"},{"why":"Introduces the linear-combination fusion principle and the 'Chorus Effect' that the paper's scoring uses.","marker":"[35]"},{"why":"Supports the claim that test collections are reusable only when pools are deep and built from effective runs.","marker":"[37]"},{"why":"Shows that a pooling depth of 10 does not guarantee reliable evaluation of systems of different types.","marker":"[44]"}],"fun_headline_variants":["Fusing personalized and neutral queries boosts passage ranking","Score-based fusion of query rewrites improves conversational search","Mixing query personalization levels improves ranking in search","Fusing rewrites at different personalization levels improves ranking","Overcoming over-personalization via retrieval fusion in search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume the human relevance judgments are fair to the fused run's documents, yet the paper's own case study shows documents retrieved by the expanded query were far less likely to be judged than documents retrieved by the human rewrite; if that pattern holds generally, the measured ranking gains could come from incomplete judgment rather than real quality.","fun_headline_variants_meta":{"raw":{"variants":["Fusing personalized and neutral queries boosts passage ranking","Score-based fusion of query rewrites improves conversational search","Mixing query personalization levels improves ranking in search","Fusing rewrites at different personalization levels improves ranking","Overcoming over-personalization via retrieval fusion in search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000994,"raw_usage":{"total_tokens":4163,"prompt_tokens":850,"completion_tokens":3313,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":3236}},"tokens_in":466,"tokens_out":3313,"duration_ms":21375,"temperature":1.0,"reasoning_tokens":3236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:19:00.327782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Judge every unassessed document in the top 10 of the fused run for all turns, recompute NDCG@10, and compare with the same recomputation for the best non-fusion run; if the fused run no longer leads, the central claim is falsified. The authors' 9-1-3 example (5 of top 20 judged for the expanded query versus 14 for the original) shows exactly the kind of incompleteness that would matter.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that large collections with shallow pooling can produce biased relevance sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the early equal-weight score-combination methods that frame the fusion design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that LLMs can build de-contextualized personalized queries and notes the lack of public training data for weight assignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the linear-combination fusion principle and the 'Chorus Effect' that the paper's scoring uses."},{"cited_title":"Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?","cited_arxiv_id":"2201.11086","evidence_quote":"Supports the claim that test collections are reusable only when pools are deep and built from effective runs."}],"review_version":1}