REVIEW 3 major objections 5 minor 23 references
IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Multi-aspect query generation beats single query rewrites
desk verdict A clean system description with a fair head-to-head comparison, but the abstract overreaches and the headline numbers lack statistical support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the MQ4CS-QR pipeline, a variation of the MQ4CS framework. It consists of a zero-shot GPT-4 prompt that takes the user utterance, conversation history, and personal knowledge base and produces up to $\phi=5$ aspect queries (queries capturing distinct sub-needs); a first-stage retriever, SPLADE, a learned sparse model that scores passages by a sparse expansion of the vocabulary; pooling of the passages retrieved by each aspect query; and a reranker, a DebertaV3 cross-encoder or an ensemble of five cross-encoders, that jointly encodes query and passage to reorder the pooled set using a single query rewrite. The key design choice is separating diversity at retrieval from focus at reranking, instead of interleaving the per-query ranked lists as the original MQ4CS does.
What would settle it
Using the released iKAT 2024 qrels, compute a per-topic paired comparison of the multi-aspect run against the single-rewrite run; if a bootstrap confidence interval for the nDCG difference straddles zero or the gap is driven by one topic, the multi-aspect advantage would not hold as a general effect.
Extended reading notes
Core claim
The central claim is that multi-aspect LLM query generation improves first-stage retrieval for conversational passage ranking, and that the improvement persists after reranking when the candidate pool is ordered by a single query rewrite. The paper's variant, MQ4CS-QR, generates up to five aspect queries from GPT-4, uses each for SPLADE retrieval, pools the retrieved passages, and reranks them with a DebertaV3 cross-encoder (or an ensemble of five cross-encoders) using an independent single rewrite. The evidence is a comparison of runs (1) and (3), which differ only in this multi-aspect versus single-query choice: the multi-aspect run reports higher Recall@100 and mAP before reranking and higher nDCG@5 and nDCG after reranking. The paper also reports that the automatic GPT-4 rewrites outperform the provided human rewrites on the same pipeline.
Load-bearing premise
The reported multi-aspect gains are differences between single runs on a 103-turn benchmark with no significance testing, so the claim assumes these gaps are real effects rather than metric noise.
Editorial extensions
If this is right
- Conversational retrieval systems can improve recall by allocating the LLM budget to several aspect queries at the retrieval stage, even when the reranker only sees a single rewrite.
- Pooling the passages retrieved by multiple aspect queries and then reranking is a more effective use of multi-aspect generation than interleaving the per-query ranked lists.
- Automatic GPT-4 rewrites can match or exceed human-written rewrites on the same pipeline, reducing the need for manual query construction in conversational search.
- Learned sparse retrieval with SPLADE is a workable first-stage for multi-aspect expansion, so the recipe does not depend on dense retrieval or an explicit fusion stage.
Reading between the lines
- Beyond the paper, the 'diversify at retrieval, focus at reranking' split is a general scheme that could transfer to query expansion in open-domain question answering or product search without changes to the underlying ranking models.
- The number of aspect queries ($\phi=5$) is a hyperparameter; a natural follow-up is to vary it and to swap GPT-4 for a smaller open-weight LLM to see whether the multi-aspect advantage persists or is tied to model scale.
- Because the PTKB supplies personal statements, the method effectively injects personalization at the query-generation stage; one testable corollary is that turns where more PTKB statements are relevant benefit more from multi-aspect generation than turns with fewer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports the authors' submissions to the TREC iKAT 2024 conversational search track. The main idea is to extend the MQ4CS framework—which generates multiple aspect-based LLM query rewrites—by using SPLADE for first-stage retrieval and a single query rewrite for reranking, rather than interleaving the ranked lists of the multiple queries. Six runs are described: four automatic (MQ4CS-QR-deberta, MQ4CS-QR-ensemble, GPT4QR-deberta, GPT4QR-bm25-QD1) and two manual (HumanQR-deberta, HumanQR-ensemble). The central empirical claim, made in Section 5.1, is that using multiple LLM-generated aspect queries for retrieval improves conversational passage ranking compared to a single query rewrite: comparing runs (1) and (3), the authors report gains of 2.3 points in Recall@100, 3.2 points in mAP, 1.5 points in nDCG@5, and 6.8 points in nDCG. The paper also claims that automatic GPT-4 rewrites outperform human rewrites and that reranker ensembling provides further gains. The contribution is positioned as a generalization of MQ4CS to the iKAT 2024 benchmark with an alternative to the original interleaving-based fusion strategy.
Significance. If the central claim holds, the paper offers a simple and reproducible recipe for conversational retrieval: generate multiple aspect queries with an LLM, retrieve with SPLADE, and rerank with a single query rewrite. The cleanest comparison—runs (1) versus (3)—isolates multiple-query versus single-query generation while holding the reranker (DebertaV3) fixed, which is a genuine strength. The inclusion of manual human-rewrite baselines and an ensemble reranker also strengthens the empirical picture. However, the significance is tempered by three issues. First, the evaluation uses only 13 topics (103 turns), and the reported gains are not accompanied by any significance test, confidence interval, or per-topic breakdown, so the differences could be within metric noise. Second, the method for merging the multiple retrieval lists before reranking is not specified, which is essential for reproduction and for attributing the gain to multi-aspect generation. Third, the claim that automatic rewrites outperform human rewrites is partially contradicted by the paper's own Table 1 on nDCG and Recall@100.
major comments (3)
- [Section 5.1, Table 1, Figure 3] The central claim that multi-aspect query generation improves retrieval is supported only by aggregate differences between runs (1) and (3) on 13 topics. No significance test, confidence interval, or per-topic error bar is reported. Figure 3 shows substantial topic-level variance (e.g., topic 8 is far below average for all runs), so the aggregate gains of 1.4–6.8 points could be driven by a few topics. Since per-topic nDCG values were evidently available to produce Figure 3, a paired test across the 13 topics (e.g., Wilcoxon signed-rank or a bootstrap interval) should be reported for the key metrics; without this, the conclusion that multi-aspect generation is effective is not statistically established.
- [Section 3.1, Figure 1] The proposed MQ4CS-QR method is underspecified at the candidate-generation step. The text says 'we propose to only use the multiple generated queries for retrieval and then re-rank all retrieved passages using a single query rewrite,' but it never states how the lists retrieved by the different generated queries are combined into the single candidate set that is fed to the reranker (e.g., union of top-k per query, score normalization, round-robin, or some other rule). Without this detail, the method cannot be reproduced, and the observed gains cannot be cleanly attributed to multi-aspect generation rather than to the fusion strategy.
- [Section 5.1, runs (3) and (5)] The text states that comparing GPT4QR-deberta (3) with HumanQR-deberta (5) shows 'how much GPT models now beat humans on the rewriting task,' but Table 1 does not support this across all metrics. HumanQR-deberta has higher nDCG (0.5496 vs. 0.5420) and higher Recall@100 (0.4263 vs. 0.4135) than GPT4QR-deberta; only nDCG@5, MRR, P@20, and mAP favor the automatic run. This overstatement should be corrected, and any claim of superiority should be limited to the specific metrics and subjected to the same significance testing recommended above.
minor comments (5)
- [Section 4.3] The statement that the authors could not use a >=2 threshold for MRR and recall because they lacked access to the qrels is confusing, since they also report receiving official per-topic nDCG from TREC (used in Figure 3). Please clarify which official scores were available and whether the reported metrics use a threshold of 1.
- [Figure 2] The x-axis is labeled 'Turn' while the text in Section 5.2 refers to 'depth' of the conversation. The wording should be aligned, and it should be clarified whether the x-axis represents absolute turn number across all conversations or a relative depth measure.
- [Table 1] The caption says 'Bold and underlined indicate the best and second-best results across both Automatic and Manual runs,' but no bold or underline appears in the table. Either add the formatting or remove the caption claim.
- [Section 4.2] The list of cross-encoders includes 'Alert' alongside 'Electra'; this appears to be a typo for 'Albert' (the reference [13] is the Albert paper).
- [Section 4.1] Run (4) is described as using 'a single query, but with the prompting strategy of MQ4CS, with φ=1.' This makes it incomparable to run (3), which uses a different single-rewrite prompt. Please clarify whether the single-rewrite prompt used in runs (1)–(3) is the same as the MQ4CS prompt with φ=1, and if not, state which prompt is used in each run.
Circularity Check
No circularity: the paper's central comparison is an empirical A/B test against the external iKAT 2024 qrels, with no fitted parameters or self-referential definitions.
full rationale
This is a TREC system-report paper rather than a derivation. The load-bearing claim, that multi-aspect query generation improves passage ranking, rests on the direct comparison of submitted runs (1) and (3) in Table 1. Both runs use GPT-4, SPLADE, and DebertaV3; the only intended difference is that run (1) retrieves with five MQ4CS aspect queries while run (3) retrieves with the single rewrite, so the comparison is not circular by construction. No parameter is fitted to the relevance labels, no quantity is defined in terms of the metric it is said to predict, and the paper does not invoke any uniqueness theorem. The self-citations to the MQ4CS framework [1] and the iKAT 2023 pipeline [2] describe the prompts and the query-generation approach, but the effectiveness of that approach is established here by the paper's own evaluated runs against an external test collection, not by the cited text. Section 4.3 explicitly states the authors lacked access to the qrels and Section 5.1 reports aggregate differences without significance tests; these are statistical-validity concerns about whether the observed 1.5-6.8 point gains are reliable on 13 topics, not evidence that a prediction reduces to its input. The manuscript even discloses metric-level counter-evidence (e.g., HumanQR-ensemble exceeds MQ4CS-QR-ensemble on nDCG@5), further showing the runs are externally constrained rather than engineered to match a target. Overall, no circular step is identifiable.
Assumptions & free parameters
free parameters (1)
- Number of generated aspect queries per turn (φ) =
5
assumptions (3)
- domain assumption Zero-shot GPT-4 query generation produces useful aspect queries for conversational context and PTKB personalization
- domain assumption SPLADE and the MS MARCO/TREC DL fine-tuned cross-encoders transfer to the ClueWeb-iKAT collection
- domain assumption The iKAT 2024 test collection (103 turns, 13 topics) with official qrels is a reliable benchmark for the observed differences
Cite this review
Pith. "Pith review of IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search." pith.science (2026). https://pith.science/paper/JR5WFQDH
@misc{pith2026241114739,
author = {Pith},
title = {Pith review of: IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/JR5WFQDH}},
note = {Machine review of arXiv:2411.14739}
}
read the original abstract
The Interactive Knowledge Assistant Track (iKAT) 2024 focuses on advancing conversational assistants, able to adapt their interaction and responses from personalized user knowledge. The track incorporates a Personal Textual Knowledge Base (PTKB) alongside Conversational AI tasks, such as passage ranking and response generation. Query Rewrite being an effective approach for resolving conversational context, we explore Large Language Models (LLMs), as query rewriters. Specifically, our submitted runs explore multi-aspect query generation using the MQ4CS framework, which we further enhance with Learned Sparse Retrieval via the SPLADE architecture, coupled with robust cross-encoder models. We also propose an alternative to the previous interleaving strategy, aggregating multiple aspects during the reranking phase. Our findings indicate that multi-aspect query generation is effective in enhancing performance when integrated with advanced retrieval and reranking models. Our results also lead the way for better personalization in Conversational Search, relying on LLMs to integrate personalization within query rewrite, and outperforming human rewrite performance.
Figures
Reference graph
Works this paper leans on
-
[1]
Zahra Abbasiantaeb, Simon Lupart, and Mohammad Aliannejadi. 2024. Gen- erating Multi-Aspect Queries for Conversational Search. arXiv preprint arXiv:2403.19302 (2024)
arXiv 2024
-
[2]
Zahra Abbasiantaeb, Chuan Meng, David Rau, Antonis Krasakis, Hossein A Rah- mani, and Mohammad Aliannejadi. 2023. LLM-based Retrieval and Generation Pipelines for TREC Interactive Knowledge Assistance Track (iKAT) 2023. (2023)
work page 2023
-
[3]
Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffrey Dal- ton, and Leif Azzopardi. 2024. TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (Washington DC, USA) (SIGIR ’24). ...
arXiv 2024
-
[4]
K Clark. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020)
arXiv 2020
-
[5]
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020. CAsT 2020: The Con- versational Assistance Track Overview. In Text Retrieval Conference . https: //api.semanticscholar.org/CorpusID:214735659
work page 2020
-
[6]
Jeffrey Dalton, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. CAsT-19: A Dataset for Conversational Information Seeking. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China)(SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 1985–1988. https://doi...
arXiv 2020
-
[8]
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant
-
[9]
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR ’21) . Association for Computing Machinery, New York, NY, USA, 2288–2292. h...
arXiv 2021
Show all 23 references
-
[10]
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 (2021). IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation ...
2021 arXiv
-
[11]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. De- berta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)
2020 arXiv
-
[12]
Kalervo Järvelin and Jaana Kekäläinen. 2000. IR Evaluation Methods for Re- trieving Highly Relevant Documents. In Proceedings of the 23rd Annual In- ternational ACM SIGIR Conference on Research and Development in Informa- tion Retrieval (Athens, Greece) (SIGIR ’00). ACM, New Y...
2000
-
[13]
Z Lan. 2019. Albert: A lite bert for self-supervised learning of language represen- tations. arXiv preprint arXiv:1909.11942 (2019)
2019 arXiv
-
[14]
Carlos Lassance and Stéphane Clinchant. 2023. Naver Labs Europe (SPLADE)@ TREC Deep Learning 2022. arXiv preprint arXiv:2302.12574 (2023)
2023 arXiv
-
[15]
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. 2024. SPLADE-v3: New baselines for SPLADE
2024
-
[16]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. InProceedings o...
2020
-
[17]
Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692 364 (2019)
2019 arXiv
-
[18]
Simon Lupart, Mohammad Aliannejadi, and Evangelos Kanoulas. 2024. DiSCo Meets LLMs: A Unified Approach for Sparse Retrieval and Contextual Distillation in Conversational Search. arXiv preprint arXiv:2410.14609 (2024)
2024 arXiv
-
[19]
Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. 2023. Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bo...
2023
-
[20]
Fengran Mo, Abbas Ghaddar, Kelong Mao, Mehdi Rezagholizadeh, Boxing Chen, Qun Liu, and Jian-Yun Nie. 2024. CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search. arXiv preprint arXiv:2406.05013 (2024)
2024 arXiv
-
[21]
Trippas, and Svitlana Vakulenko
Paul Owoicho, Jeffrey Dalton, Mohammad Aliannejadi, Leif Azzopardi, Johanne R. Trippas, and Svitlana Vakulenko. 2022. TREC CAsT 2022: Going Beyond User Ask and System Retrieve with Initiative and Response Generation. In Text Retrieval Conference. https://api.semanticscholar.or...
2022
-
[1225]
https://doi.org/10.18653/v1/2023.findings-emnlp.86
2023 doi
-
[2021]
arXiv preprint arXiv:2109.10086 (2021)
SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021)
2021 arXiv
-
[2022]
In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22)
From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22). Association for Computing Machinery, New ...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.